The bit_reader and Huffman modules were ported directly using the provided memory manager. We created rust-alloc-no-stdlib for this purpose.Ģ. This ensured we could add SECCOMP support later and could operate without the stdlib. A foundational memory manager with a malloc-like interface needed to be created to abstract away any stdlib access. Porting Brotli from C to Rust required a couple of steps to ensure it met our goals:ġ. Finally, we enter the process into the secure computing (SECCOMP) mode, disabling any system calls except for read, write, sigreturn and exit. After the virtual memory is allocated, we enable a timer using the alarm syscall, to avoid a runaway process that never returns control. This allows us to put an upper bound on the memory we would allow for the decode of a single 4MB block. We also went one step further and wrote our own Rust memory allocator that can be used to allocate memory in the standard way using Boxes, or from a fixed size allocation on the heap, or even a pool on the stack. We created rust-brotli, a direct port of the C decompressor into safe Rust. At Dropbox, many of our services are actually memory bound, so this is a key advantage over a garbage collected language.
#LOSSLESS VIDEO COMPRESSION CODE#
That means that code written in Rust has the same memory requirements as the equivalent code written in C. It also has sufficient performance for our needs. The Rust programing language fits the bill perfectly: it’s a language that promises memory safety without garbage collection, concurrency without data races, and abstractions without overhead. array out of bounds access) or nondeterministic code (eg reading uninitialized memory), so therefore we can trust the code to repeatably produce the same output without any security risks. This is because such a language would prevent us from executing unsafe code (eg. By writing a new Brotli decompressor in a language that is safe and deterministic, we only needed to analyze the language, not all the code written in it. Operating at Dropbox scale, we need to guarantee the security of our data, so our approach was to break down the problem into components. It could be both secure and deterministic, but there is simply too much code to reason through a mathematical proof of this hypothesis. Since the Brotli decompressor consists of a substantial amount of C code written by human beings, it is possibly neither deterministic nor safe and secure against carefully crafted hostile data.
Unfortunately, the compressor supplied by the Brotli project only has the third property: it is very fast.
#LOSSLESS VIDEO COMPRESSION FULL#
With these properties we can accept any arbitrary bytes from a client and have full knowledge that those bytes factually represent the file data. it must be deterministic-the same bytes must result in the same output, it must be safe and secure, even against bytes crafted by modified or hostile clients,Ģ.
Once the files have been uploaded, they need to be durably persisted as long as the user wishes, and at a moment’s notice they may need to be restored to their original bits exactly in a repeatable, secure way.įor Dropbox, any decompressor must exhibit three properties:ġ.
0 kommentar(er)
