Industry Buzz

Tips to Share Your Insights and Experiences on LinkedIn

LinkedIn Official Blog -

In this new world of social distancing, it's easy to feel alone. That’s why it’s more important than ever to keep connected with one another and identify ways you can help by offering your experiences. We’ve seen sharing on LinkedIn increase, with members discussing topics like remote working, online learning and collaboration, indicating there’s a desire and need to understand how people are adjusting to new routines.  No matter what industry you’re in or where you are in your career,  it’s a... .

5 Quick Website Updates You Can Do in 30 Minutes or Less

HostGator Blog -

The post 5 Quick Website Updates You Can Do in 30 Minutes or Less appeared first on HostGator Blog. This article is part of HostGator’s Web Pros Series. In this series, we feature articles from our team of experts here at HostGator. Our Product Managers, Linux Administrators, Marketers, and Tech Support engineers share their best tips for getting the most out of your website.  So you’re working from home now. The kids are finally busy working on a craft project and the barking dog fell asleep.  You’ve got less than an hour before the “you know what” hits the fan again. And you want to do something that can quickly move the needle for your business.  Here are 5 quick website updates you can make in 30 minutes or less.  Note: You can’t do all of these things in 30 minutes. But one task at a time, with 30 minutes at a time, you’ll get it done! 1. Update your business hours on your website & on Google My Business. Maybe your city has gone on lock down that required your physical location to go on revised hours. Or maybe you’re temporarily closed on a week-by-week basis. Make sure your customers and clients know your new, current operating hours by updating your contact page.  Consider adding a “current as of X date” tag or a note that calls out “our current hours during COVID-19” so customers know they are truly seeing your current, updated hours.  You may also consider a call out banner on the homepage so your customers can easily find this information.  Next, update your temporary business hours on your Google My Business listing. This is important because it will populate the store listing that customers see when searching for your physical location on Google Maps app.  Recently I was out for a late night run to the local drug store and looked up the hours on Google Maps. The hours listed regular hours and then this week’s hours. Sweet! They’re still open 24 hours. I arrive, and JK, the store is closed and there’s a sign on the door with early closing hours. Super frustrating! There are three ways you can update your business hours on Google My Business. Check out the “holiday hours” option.  2. Review & update your metadata.  Meta what? This is a search engine optimization thing. And we all need those positive SEO vibes right now.  Metadata is what appears in the search engine results when your website comes up for a query. Like in the example below.  If you don’t have metadata set up, the search engine will create this for you. But it will likely not be what you want and the meta description will be truncated by character limit.  If you’re on a WordPress website, you can tackle this 30 minute SEO task by setting up the Yoast SEO plugin. Yoast makes it easy to change the page title and meta description directly within the editor of each post.  Another added bonus – you don’t have to remember the latest standards for metadescription length. Yoast will highlight the text in red, yellow, or green to let you know how much space is remaining.  Got more than 30 minutes? Watch our Intro to SEO webinar to learn more about metadata.  3. Add your phone number (or preferred contact method) to your site header.  Now more than ever, your website should be easy to use so people can quickly find the information they need.  Consider adding a phone number or your preferred method of contact to your website header so people can easily see it. If you have a form or an informational update they should see first before contacting you, draw attention to that as well.  Southwest Airlines used this method by listing important links in the top header and a “High Call Volumes” notice so customers know what to expect. I was quickly able to find their policy about cancelling a flight in the “information for you” links, and didn’t even have to call! I’m sure they’ll appreciate one less phone call in these crazy times.    4. Write a blog post and share it on social media.  A blog post is a great way to show that your business is still up and running. Need an idea for a quick post? Write about your top selling item of the year.  Got an item or service that could help people during these challenging times? Write a product highlight in a blog post and share it on your social channels. This doesn’t have to be long, but make it informative and helpful.   5. Review and update your navigation menu.  Now is a great time to review the buttons and options in your navigation menu. Are these items still relevant to your customers or shoppers, given the current environment?  On the HostGator blog, we’re working hard to pump out a lot of content to support our customers during the new state of work. I wanted to be able to point website visitors to one link where they could see all this great content. And so it doesn’t get lost in our steady stream of new articles that are publishing each day.  So I added “COVID-19 Business Tips” to the main navigation. Now this content – which is currently our priority focus – is highlighted and easy for people to find.  While I was in there, I also changed “Resource Library” to “Webinars & More.” This seemed more descriptive of the content and would catch attention while people are coming to our site to learn new things about going online.  Wrapping Up… When you want to make some meaningful changes to your website on a time crunch, these ideas will get you started. Now, hurry up! Pick your fave and get it done before the dog wakes up! Find the post on the HostGator Blog

Why Website Architecture Matters To SEO

Pickaweb Blog -

Without some kind of organised structure, websites would become a random assortment of web pages that would be impossible for both users to navigate. What ties these pages together is a URL hierarchy that is manifested through the websites navigation, usually in the form of nav bars, drop down menus and icons and links to The post Why Website Architecture Matters To SEO appeared first on Pickaweb.

Speeding up Linux disk encryption

CloudFlare Blog -

Data encryption at rest is a must-have for any modern Internet company. Many companies, however, don't encrypt their disks, because they fear the potential performance penalty caused by encryption overhead. Encrypting data at rest is vital for Cloudflare with more than 200 data centres across the world. In this post, we will investigate the performance of disk encryption on Linux and explain how we made it at least two times faster for ourselves and our customers! Encrypting data at rest When it comes to encrypting data at rest there are several ways it can be implemented on a modern operating system (OS). Available techniques are tightly coupled with a typical OS storage stack. A simplified version of the storage stack and encryption solutions can be found on the diagram below: On the top of the stack are applications, which read and write data in files (or streams). The file system in the OS kernel keeps track of which blocks of the underlying block device belong to which files and translates these file reads and writes into block reads and writes, however the hardware specifics of the underlying storage device is abstracted away from the filesystem. Finally, the block subsystem actually passes the block reads and writes to the underlying hardware using appropriate device drivers. The concept of the storage stack is actually similar to the well-known network OSI model, where each layer has a more high-level view of the information and the implementation details of the lower layers are abstracted away from the upper layers. And, similar to the OSI model, one can apply encryption at different layers (think about TLS vs IPsec or a VPN). For data at rest we can apply encryption either at the block layers (either in hardware or in software) or at the file level (either directly in applications or in the filesystem). Block vs file encryption Generally, the higher in the stack we apply encryption, the more flexibility we have. With application level encryption the application maintainers can apply any encryption code they please to any particular data they need. The downside of this approach is they actually have to implement it themselves and encryption in general is not very developer-friendly: one has to know the ins and outs of a specific cryptographic algorithm, properly generate keys, nonces, IVs etc. Additionally, application level encryption does not leverage OS-level caching and Linux page cache in particular: each time the application needs to use the data, it has to either decrypt it again, wasting CPU cycles, or implement its own decrypted “cache”, which introduces more complexity to the code. File system level encryption makes data encryption transparent to applications, because the file system itself encrypts the data before passing it to the block subsystem, so files are encrypted regardless if the application has crypto support or not. Also, file systems can be configured to encrypt only a particular directory or have different keys for different files. This flexibility, however, comes at a cost of a more complex configuration. File system encryption is also considered less secure than block device encryption as only the contents of the files are encrypted. Files also have associated metadata, like file size, the number of files, the directory tree layout etc., which are still visible to a potential adversary. Encryption down at the block layer (often referred to as disk encryption or full disk encryption) also makes data encryption transparent to applications and even whole file systems. Unlike file system level encryption it encrypts all data on the disk including file metadata and even free space. It is less flexible though - one can only encrypt the whole disk with a single key, so there is no per-directory, per-file or per-user configuration. From the crypto perspective, not all cryptographic algorithms can be used as the block layer doesn't have a high-level overview of the data anymore, so it needs to process each block independently. Most common algorithms require some sort of block chaining to be secure, so are not applicable to disk encryption. Instead, special modes were developed just for this specific use-case. So which layer to choose? As always, it depends... Application and file system level encryption are usually the preferred choice for client systems because of the flexibility. For example, each user on a multi-user desktop may want to encrypt their home directory with a key they own and leave some shared directories unencrypted. On the contrary, on server systems, managed by SaaS/PaaS/IaaS companies (including Cloudflare) the preferred choice is configuration simplicity and security - with full disk encryption enabled any data from any application is automatically encrypted with no exceptions or overrides. We believe that all data needs to be protected without sorting it into "important" vs "not important" buckets, so the selective flexibility the upper layers provide is not needed. Hardware vs software disk encryption When encrypting data at the block layer it is possible to do it directly in the storage hardware, if the hardware supports it. Doing so usually gives better read/write performance and consumes less resources from the host. However, since most hardware firmware is proprietary, it does not receive as much attention and review from the security community. In the past this led to flaws in some implementations of hardware disk encryption, which render the whole security model useless. Microsoft, for example, started to prefer software-based disk encryption since then. We didn't want to put our data and our customers' data to the risk of using potentially insecure solutions and we strongly believe in open-source. That's why we rely only on software disk encryption in the Linux kernel, which is open and has been audited by many security professionals across the world. Linux disk encryption performance We aim not only to save bandwidth costs for our customers, but to deliver content to Internet users as fast as possible. At one point we noticed that our disks were not as fast as we would like them to be. Some profiling as well as a quick A/B test pointed to Linux disk encryption. Because not encrypting the data (even if it is supposed-to-be a public Internet cache) is not a sustainable option, we decided to take a closer look into Linux disk encryption performance. Device mapper and dm-crypt Linux implements transparent disk encryption via a dm-crypt module and dm-crypt itself is part of device mapper kernel framework. In a nutshell, the device mapper allows pre/post-process IO requests as they travel between the file system and the underlying block device. dm-crypt in particular encrypts "write" IO requests before sending them further down the stack to the actual block device and decrypts "read" IO requests before sending them up to the file system driver. Simple and easy! Or is it? Benchmarking setup For the record, the numbers in this post were obtained by running specified commands on an idle Cloudflare G9 server out of production. However, the setup should be easily reproducible on any modern x86 laptop. Generally, benchmarking anything around a storage stack is hard because of the noise introduced by the storage hardware itself. Not all disks are created equal, so for the purpose of this post we will use the fastest disks available out there - that is no disks. Instead Linux has an option to emulate a disk directly in RAM. Since RAM is much faster than any persistent storage, it should introduce little bias in our results. The following command creates a 4GB ramdisk: $ sudo modprobe brd rd_nr=1 rd_size=4194304 $ ls /dev/ram0 Now we can set up a dm-crypt instance on top of it thus enabling encryption for the disk. First, we need to generate the disk encryption key, "format" the disk and specify a password to unlock the newly generated key. $ fallocate -l 2M crypthdr.img $ sudo cryptsetup luksFormat /dev/ram0 --header crypthdr.img WARNING! ======== This will overwrite data on crypthdr.img irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: Those who are familiar with LUKS/dm-crypt might have noticed we used a LUKS detached header here. Normally, LUKS stores the password-encrypted disk encryption key on the same disk as the data, but since we want to compare read/write performance between encrypted and unencrypted devices, we might accidentally overwrite the encrypted key during our benchmarking later. Keeping the encrypted key in a separate file avoids this problem for the purposes of this post. Now, we can actually "unlock" the encrypted device for our testing: $ sudo cryptsetup open --header crypthdr.img /dev/ram0 encrypted-ram0 Enter passphrase for /dev/ram0: $ ls /dev/mapper/encrypted-ram0 /dev/mapper/encrypted-ram0 At this point we can now compare the performance of encrypted vs unencrypted ramdisk: if we read/write data to /dev/ram0, it will be stored in plaintext. Likewise, if we read/write data to /dev/mapper/encrypted-ram0, it will be decrypted/encrypted on the way by dm-crypt and stored in ciphertext. It's worth noting that we're not creating any file system on top of our block devices to avoid biasing results with a file system overhead. Measuring throughput When it comes to storage testing/benchmarking Flexible I/O tester is the usual go-to solution. Let's simulate simple sequential read/write load with 4K block size on the ramdisk without encryption: $ sudo fio --filename=/dev/ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=plain plain: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1 fio-2.16 Starting 1 process ... Run status group 0 (all jobs): READ: io=21013MB, aggrb=1126.5MB/s, minb=1126.5MB/s, maxb=1126.5MB/s, mint=18655msec, maxt=18655msec WRITE: io=21023MB, aggrb=1126.1MB/s, minb=1126.1MB/s, maxb=1126.1MB/s, mint=18655msec, maxt=18655msec Disk stats (read/write): ram0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% The above command will run for a long time, so we just stop it after a while. As we can see from the stats, we're able to read and write roughly with the same throughput around 1126 MB/s. Let's repeat the test with the encrypted ramdisk: $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt crypt: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1 fio-2.16 Starting 1 process ... Run status group 0 (all jobs): READ: io=1693.7MB, aggrb=150874KB/s, minb=150874KB/s, maxb=150874KB/s, mint=11491msec, maxt=11491msec WRITE: io=1696.4MB, aggrb=151170KB/s, minb=151170KB/s, maxb=151170KB/s, mint=11491msec, maxt=11491msec Whoa, that's a drop! We only get ~147 MB/s now, which is more than 7 times slower! And this is on a totally idle machine! Maybe, crypto is just slow The first thing we considered is to ensure we use the fastest crypto. cryptsetup allows us to benchmark all the available crypto implementations on the system to select the best one: $ sudo cryptsetup benchmark # Tests are approximate using memory only (no storage IO). PBKDF2-sha1 1340890 iterations per second for 256-bit key PBKDF2-sha256 1539759 iterations per second for 256-bit key PBKDF2-sha512 1205259 iterations per second for 256-bit key PBKDF2-ripemd160 967321 iterations per second for 256-bit key PBKDF2-whirlpool 720175 iterations per second for 256-bit key # Algorithm | Key | Encryption | Decryption aes-cbc 128b 969.7 MiB/s 3110.0 MiB/s serpent-cbc 128b N/A N/A twofish-cbc 128b N/A N/A aes-cbc 256b 756.1 MiB/s 2474.7 MiB/s serpent-cbc 256b N/A N/A twofish-cbc 256b N/A N/A aes-xts 256b 1823.1 MiB/s 1900.3 MiB/s serpent-xts 256b N/A N/A twofish-xts 256b N/A N/A aes-xts 512b 1724.4 MiB/s 1765.8 MiB/s serpent-xts 512b N/A N/A twofish-xts 512b N/A N/A It seems aes-xts with a 256-bit data encryption key is the fastest here. But which one are we actually using for our encrypted ramdisk? $ sudo dmsetup table /dev/mapper/encrypted-ram0 0 8388608 crypt aes-xts-plain64 0000000000000000000000000000000000000000000000000000000000000000 0 1:0 0 We do use aes-xts with a 256-bit data encryption key (count all the zeroes conveniently masked by dmsetup tool - if you want to see the actual bytes, add the --showkeys option to the above command). The numbers do not add up however: cryptsetup benchmark tells us above not to rely on the results, as "Tests are approximate using memory only (no storage IO)", but that is exactly how we've set up our experiment using the ramdisk. In a somewhat worse case (assuming we're reading all the data and then encrypting/decrypting it sequentially with no parallelism) doing back-of-the-envelope calculation we should be getting around (1126 * 1823) / (1126 + 1823) =~696 MB/s, which is still quite far from the actual 147 * 2 = 294 MB/s (total for reads and writes). dm-crypt performance flags While reading the cryptsetup man page we noticed that it has two options prefixed with --perf-, which are probably related to performance tuning. The first one is --perf-same_cpu_crypt with a rather cryptic description: Perform encryption using the same cpu that IO was submitted on. The default is to use an unbound workqueue so that encryption work is automatically balanced between available CPUs. This option is only relevant for open action. So we enable the option $ sudo cryptsetup close encrypted-ram0 $ sudo cryptsetup open --header crypthdr.img --perf-same_cpu_crypt /dev/ram0 encrypted-ram0 Note: according to the latest man page there is also a cryptsetup refresh command, which can be used to enable these options live without having to "close" and "re-open" the encrypted device. Our cryptsetup however didn't support it yet. Verifying if the option has been really enabled: $ sudo dmsetup table encrypted-ram0 0 8388608 crypt aes-xts-plain64 0000000000000000000000000000000000000000000000000000000000000000 0 1:0 0 1 same_cpu_crypt Yes, we can now see same_cpu_crypt in the output, which is what we wanted. Let's rerun the benchmark: $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt crypt: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1 fio-2.16 Starting 1 process ... Run status group 0 (all jobs): READ: io=1596.6MB, aggrb=139811KB/s, minb=139811KB/s, maxb=139811KB/s, mint=11693msec, maxt=11693msec WRITE: io=1600.9MB, aggrb=140192KB/s, minb=140192KB/s, maxb=140192KB/s, mint=11693msec, maxt=11693msec Hmm, now it is ~136 MB/s which is slightly worse than before, so no good. What about the second option --perf-submit_from_crypt_cpus: Disable offloading writes to a separate thread after encryption. There are some situations where offloading write bios from the encryption threads to a single thread degrades performance significantly. The default is to offload write bios to the same thread. This option is only relevant for open action. Maybe, we are in the "some situation" here, so let's try it out: $ sudo cryptsetup close encrypted-ram0 $ sudo cryptsetup open --header crypthdr.img --perf-submit_from_crypt_cpus /dev/ram0 encrypted-ram0 Enter passphrase for /dev/ram0: $ sudo dmsetup table encrypted-ram0 0 8388608 crypt aes-xts-plain64 0000000000000000000000000000000000000000000000000000000000000000 0 1:0 0 1 submit_from_crypt_cpus And now the benchmark: $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt crypt: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1 fio-2.16 Starting 1 process ... Run status group 0 (all jobs): READ: io=2066.6MB, aggrb=169835KB/s, minb=169835KB/s, maxb=169835KB/s, mint=12457msec, maxt=12457msec WRITE: io=2067.7MB, aggrb=169965KB/s, minb=169965KB/s, maxb=169965KB/s, mint=12457msec, maxt=12457msec ~166 MB/s, which is a bit better, but still not good... Asking the community Being desperate we decided to seek support from the Internet and posted our findings to the dm-crypt mailing list, but the response we got was not very encouraging: If the numbers disturb you, then this is from lack of understanding on your side. You are probably unaware that encryption is a heavy-weight operation... We decided to make a scientific research on this topic by typing "is encryption expensive" into Google Search and one of the top results, which actually contains meaningful measurements, is... our own post about cost of encryption, but in the context of TLS! This is a fascinating read on its own, but the gist is: modern crypto on modern hardware is very cheap even at Cloudflare scale (doing millions of encrypted HTTP requests per second). In fact, it is so cheap that Cloudflare was the first provider to offer free SSL/TLS for everyone. Digging into the source code When trying to use the custom dm-crypt options described above we were curious why they exist in the first place and what is that "offloading" all about. Originally we expected dm-crypt to be a simple "proxy", which just encrypts/decrypts data as it flows through the stack. Turns out dm-crypt does more than just encrypting memory buffers and a (simplified) IO traverse path diagram is presented below: When the file system issues a write request, dm-crypt does not process it immediately - instead it puts it into a workqueue named "kcryptd". In a nutshell, a kernel workqueue just schedules some work (encryption in this case) to be performed at some later time, when it is more convenient. When "the time" comes, dm-crypt sends the request to Linux Crypto API for actual encryption. However, modern Linux Crypto API is asynchronous as well, so depending on which particular implementation your system will use, most likely it will not be processed immediately, but queued again for "later time". When Linux Crypto API will finally do the encryption, dm-crypt may try to sort pending write requests by putting each request into a red-black tree. Then a separate kernel thread again at "some time later" actually takes all IO requests in the tree and sends them down the stack. Now for read requests: this time we need to get the encrypted data first from the hardware, but dm-crypt does not just ask for the driver for the data, but queues the request into a different workqueue named "kcryptd_io". At some point later, when we actually have the encrypted data, we schedule it for decryption using the now familiar "kcryptd" workqueue. "kcryptd" will send the request to Linux Crypto API, which may decrypt the data asynchronously as well. To be fair the request does not always traverse all these queues, but the important part here is that write requests may be queued up to 4 times in dm-crypt and read requests up to 3 times. At this point we were wondering if all this extra queueing can cause any performance issues. For example, there is a nice presentation from Google about the relationship between queueing and tail latency. One key takeaway from the presentation is: A significant amount of tail latency is due to queueing effects So, why are all these queues there and can we remove them? Git archeology No-one writes more complex code just for fun, especially for the OS kernel. So all these queues must have been put there for a reason. Luckily, the Linux kernel source is managed by git, so we can try to retrace the changes and the decisions around them. The "kcryptd" workqueue was in the source since the beginning of the available history with the following comment: Needed because it would be very unwise to do decryption in an interrupt context, so bios returning from read requests get queued here. So it was for reads only, but even then - why do we care if it is interrupt context or not, if Linux Crypto API will likely use a dedicated thread/queue for encryption anyway? Well, back in 2005 Crypto API was not asynchronous, so this made perfect sense. In 2006 dm-crypt started to use the "kcryptd" workqueue not only for encryption, but for submitting IO requests: This patch is designed to help dm-crypt comply with the new constraints imposed by the following patch in -mm: md-dm-reduce-stack-usage-with-stacked-block-devices.patch It seems the goal here was not to add more concurrency, but rather reduce kernel stack usage, which makes sense again as the kernel has a common stack across all the code, so it is a quite limited resource. It is worth noting, however, that the Linux kernel stack has been expanded in 2014 for x86 platforms, so this might not be a problem anymore. A first version of "kcryptd_io" workqueue was added in 2007 with the intent to avoid: starvation caused by many requests waiting for memory allocation... The request processing was bottlenecking on a single workqueue here, so the solution was to add another one. Makes sense. We are definitely not the first ones experiencing performance degradation because of extensive queueing: in 2011 a change was introduced to conditionally revert some of the queueing for read requests: If there is enough memory, code can directly submit bio instead queuing this operation in a separate thread. Unfortunately, at that time Linux kernel commit messages were not as verbose as today, so there is no performance data available. In 2015 dm-crypt started to sort writes in a separate "dmcrypt_write" thread before sending them down the stack: On a multiprocessor machine, encryption requests finish in a different order than they were submitted. Consequently, write requests would be submitted in a different order and it could cause severe performance degradation. It does make sense as sequential disk access used to be much faster than the random one and dm-crypt was breaking the pattern. But this mostly applies to spinning disks, which were still dominant in 2015. It may not be as important with modern fast SSDs (including NVME SSDs). Another part of the commit message is worth mentioning: particular it enables IO schedulers like CFQ to sort more effectively... It mentions the performance benefits for the CFQ IO scheduler, but Linux schedulers have improved since then to the point that CFQ scheduler has been removed from the kernel in 2018. The same patchset replaces the sorting list with a red-black tree: In theory the sorting should be performed by the underlying disk scheduler, however, in practice the disk scheduler only accepts and sorts a finite number of requests. To allow the sorting of all requests, dm-crypt needs to implement its own sorting. The overhead associated with rbtree-based sorting is considered negligible so it is not used conditionally. All that make sense, but it would be nice to have some backing data. Interestingly, in the same patchset we see the introduction of our familiar "submit_from_crypt_cpus" option: There are some situations where offloading write bios from the encryption threads to a single thread degrades performance significantly Overall, we can see that every change was reasonable and needed, however things have changed since then: hardware became faster and smarter Linux resource allocation was revisited coupled Linux subsystems were rearchitected And many of the design choices above may not be applicable to modern Linux. The "clean-up" Based on the research above we decided to try to remove all the extra queueing and asynchronous behaviour and revert dm-crypt to its original purpose: simply encrypt/decrypt IO requests as they pass through. But for the sake of stability and further benchmarking we ended up not removing the actual code, but rather adding yet another dm-crypt option, which bypasses all the queues/threads, if enabled. The flag allows us to switch between the current and new behaviour at runtime under full production load, so we can easily revert our changes should we see any side-effects. The resulting patch can be found on the Cloudflare GitHub Linux repository. Synchronous Linux Crypto API From the diagram above we remember that not all queueing is implemented in dm-crypt. Modern Linux Crypto API may also be asynchronous and for the sake of this experiment we want to eliminate queues there as well. What does "may be" mean, though? The OS may contain different implementations of the same algorithm (for example, hardware-accelerated AES-NI on x86 platforms and generic C-code AES implementations). By default the system chooses the "best" one based on the configured algorithm priority. dm-crypt allows overriding this behaviour and request a particular cipher implementation using the capi: prefix. However, there is one problem. Let us actually check the available AES-XTS (this is our disk encryption cipher, remember?) implementations on our system: $ grep -A 11 'xts(aes)' /proc/crypto name : xts(aes) driver : xts(ecb(aes-generic)) module : kernel priority : 100 refcnt : 7 selftest : passed internal : no type : skcipher async : no blocksize : 16 min keysize : 32 max keysize : 64 -- name : __xts(aes) driver : cryptd(__xts-aes-aesni) module : cryptd priority : 451 refcnt : 1 selftest : passed internal : yes type : skcipher async : yes blocksize : 16 min keysize : 32 max keysize : 64 -- name : xts(aes) driver : xts-aes-aesni module : aesni_intel priority : 401 refcnt : 1 selftest : passed internal : no type : skcipher async : yes blocksize : 16 min keysize : 32 max keysize : 64 -- name : __xts(aes) driver : __xts-aes-aesni module : aesni_intel priority : 401 refcnt : 7 selftest : passed internal : yes type : skcipher async : no blocksize : 16 min keysize : 32 max keysize : 64 We want to explicitly select a synchronous cipher from the above list to avoid queueing effects in threads, but the only two supported are xts(ecb(aes-generic)) (the generic C implementation) and __xts-aes-aesni (the x86 hardware-accelerated implementation). We definitely want the latter as it is much faster (we're aiming for performance here), but it is suspiciously marked as internal (see internal: yes). If we check the source code: Mark a cipher as a service implementation only usable by another cipher and never by a normal user of the kernel crypto API So this cipher is meant to be used only by other wrapper code in the Crypto API and not outside it. In practice this means, that the caller of the Crypto API needs to explicitly specify this flag, when requesting a particular cipher implementation, but dm-crypt does not do it, because by design it is not part of the Linux Crypto API, rather an "external" user. We already patch the dm-crypt module, so we could as well just add the relevant flag. However, there is another problem with AES-NI in particular: x86 FPU. "Floating point" you say? Why do we need floating point math to do symmetric encryption which should only be about bit shifts and XOR operations? We don't need the math, but AES-NI instructions use some of the CPU registers, which are dedicated to the FPU. Unfortunately the Linux kernel does not always preserve these registers in interrupt context for performance reasons (saving/restoring FPU is expensive). But dm-crypt may execute code in interrupt context, so we risk corrupting some other process data and we go back to "it would be very unwise to do decryption in an interrupt context" statement in the original code. Our solution to address the above was to create another somewhat "smart" Crypto API module. This module is synchronous and does not roll its own crypto, but is just a "router" of encryption requests: if we can use the FPU (and thus AES-NI) in the current execution context, we just forward the encryption request to the faster, "internal" __xts-aes-aesni implementation (and we can use it here, because now we are part of the Crypto API) otherwise, we just forward the encryption request to the slower, generic C-based xts(ecb(aes-generic)) implementation Using the whole lot Let's walk through the process of using it all together. The first step is to grab the patches and recompile the kernel (or just compile dm-crypt and our xtsproxy modules). Next, let's restart our IO workload in a separate terminal, so we can make sure we can reconfigure the kernel at runtime under load: $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt crypt: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1 fio-2.16 Starting 1 process ... In the main terminal make sure our new Crypto API module is loaded and available: $ sudo modprobe xtsproxy $ grep -A 11 'xtsproxy' /proc/crypto driver : xts-aes-xtsproxy module : xtsproxy priority : 0 refcnt : 0 selftest : passed internal : no type : skcipher async : no blocksize : 16 min keysize : 32 max keysize : 64 ivsize : 16 chunksize : 16 Reconfigure the encrypted disk to use our newly loaded module and enable our patched dm-crypt flag (we have to use low-level dmsetup tool and cryptsetup obviously is not aware of our modifications): $ sudo dmsetup table encrypted-ram0 --showkeys | sed 's/aes-xts-plain64/capi:xts-aes-xtsproxy-plain64/' | sed 's/$/ 1 force_inline/' | sudo dmsetup reload encrypted-ram0 We just "loaded" the new configuration, but for it to take effect, we need to suspend/resume the encrypted device: $ sudo dmsetup suspend encrypted-ram0 && sudo dmsetup resume encrypted-ram0 And now observe the result. We may go back to the other terminal running the fio job and look at the output, but to make things nicer, here's a snapshot of the observed read/write throughput in Grafana: Wow, we have more than doubled the throughput! With the total throughput of ~640 MB/s we're now much closer to the expected ~696 MB/s from above. What about the IO latency? (The await statistic from the iostat reporting tool): The latency has been cut in half as well! To production So far we have been using a synthetic setup with some parts of the full production stack missing, like file systems, real hardware and most importantly, production workload. To ensure we’re not optimising imaginary things, here is a snapshot of the production impact these changes bring to the caching part of our stack: This graph represents a three-way comparison of the worst-case response times (99th percentile) for a cache hit in one of our servers. The green line is from a server with unencrypted disks, which we will use as baseline. The red line is from a server with encrypted disks with the default Linux disk encryption implementation and the blue line is from a server with encrypted disks and our optimisations enabled. As we can see the default Linux disk encryption implementation has a significant impact on our cache latency in worst case scenarios, whereas the patched implementation is indistinguishable from not using encryption at all. In other words the improved encryption implementation does not have any impact at all on our cache response speed, so we basically get it for free! That’s a win! We're just getting started This post shows how an architecture review can double the performance of a system. Also we reconfirmed that modern cryptography is not expensive and there is usually no excuse not to protect your data. We are going to submit this work for inclusion in the main kernel source tree, but most likely not in its current form. Although the results look encouraging we have to remember that Linux is a highly portable operating system: it runs on powerful servers as well as small resource constrained IoT devices and on many other CPU architectures as well. The current version of the patches just optimises disk encryption for a particular workload on a particular architecture, but Linux needs a solution which runs smoothly everywhere. That said, if you think your case is similar and you want to take advantage of the performance improvements now, you may grab the patches and hopefully provide feedback. The runtime flag makes it easy to toggle the functionality on the fly and a simple A/B test may be performed to see if it benefits any particular case or setup. These patches have been running across our wide network of more than 200 data centres on five generations of hardware, so can be reasonably considered stable. Enjoy both performance and security from Cloudflare for all!

Ways To Refresh Your Email Marketing Strategies In 2020

Pickaweb Blog -

Every business uses email to communicate with their clients. There are Skype, Slack, and other communication channels, but businesses still revolve around email. Email Marketing is a part of digital marketing. It helps to bring your product or service knowledge to your reader’s attention, and generate leads through emails. Nowadays, social media marketing, quick tips The post Ways To Refresh Your Email Marketing Strategies In 2020 appeared first on Pickaweb.

How to Create LinkedIn Sponsored Content Ads: A Walkthrough

Social Media Examiner -

Want to try LinkedIn advertising but don’t know how to get started? Wondering which ad type you should try first? In this article, you’ll learn why LinkedIn sponsored content ads are perfect for your first campaign. You’ll also find a walkthrough for setting up and launching your own ad campaign for website visits. You’ll discover […] The post How to Create LinkedIn Sponsored Content Ads: A Walkthrough appeared first on Social Media Marketing | Social Media Examiner.

WordPress 5.4 RC4 News -

The fourth release candidate for WordPress 5.4 is live! WordPress 5.4 is currently scheduled to land on March 31 2020, and we need your help to get there—if you haven’t tried 5.4 yet, now is the time! You can test the WordPress 5.4 release candidate in two ways: Try the WordPress Beta Tester plugin (choose the “bleeding edge nightlies” option)Or download the release candidate here (zip). For details about what to expect in WordPress 5.4, please see the first release candidate post. RC4 commits the new About page and updates the editor packages. Plugin and Theme Developers Please test your plugins and themes against WordPress 5.4 and update the Tested up to version in the readme to 5.4. The priority in testing is compatibility. If you find issues, please be sure to post to the support forums so we can figure them out before the final release. The WordPress 5.4 Field Guide is also out! It’s your source for details on all the major changes. How to Help Do you speak a language besides English? Help us translate WordPress into more than 100 languages! If you think you’ve found a bug, you can post to the Alpha/Beta area in the support forums. We’d love to hear from you! If you’re comfortable writing a reproducible bug report, file one on WordPress Trac, where you can also find a list of known bugs.

Coming Together During COVID-19

cPanel Blog -

First, we want to assure you that we are here to support you and your business during this time of uncertainty. We are implementing our tested plans to keep operations up and running, and our goal remains to continue with uninterrupted operations and service to our Partners and customers. Adjusting to the current COVID-19 situation has been a task that everyone across the world has had to deal with. cPanel has been no different, and we ...

Employees Working Remote?Cybersecurity Checklist for Small Business Owners

HostGator Blog -

The post Employees Working Remote?Cybersecurity Checklist for Small Business Owners appeared first on HostGator Blog. Millions of Americans are “working from home” due to the COVID-19  pandemic. However, the sudden transition to working at home leaves many businesses vulnerable to cybersecurity threats.  But new cybersecurity threats could shut businesses down, hold their data for ransom, and cost them big in stolen funds. As if you didn’t already have enough to worry about, am I right!?  Big businesses have the resources and IT staff to help secure their remote workforce. But most smaller organizations don’t—and hackers know it. If your SMB recently transitioned to remote work or is about to, here’s a cybersecurity checklist to keep your business safe.  First, why should SMBs worry about remote work security right now? There are 3 big reasons why SMBs need to focus on cybersecurity as their people work from home.  With every new device and network used to access your company’s data, your attack surface grows. What that means is that there are more potential ways for hackers to break into your systems. For example, if your payroll manager logs into your accounts from a phone over a public Wi-Fi network, hackers could steal their login credentials and get into your accounts, too.Cybercriminals profit from chaos and stress. Our current reality delivers both, which is why fraudsters are launching all kinds of coronavirus-related scams aimed at businesses, consumers, even hospitals. When your administrative assistant gets an urgent email from you directing them to make an online donation to a COVID-19 charity in the company’s name, they might do it without question—without realizing the email came from a scammer impersonating you to steal company funds.Most of us are less cybersecure at home than at work. Even if your cybersecurity game is perfect in your workplace, out-of-date or unpatched software on an employee’s home computer could give hackers the security gap they need to worm their way into your business.  OK, so how can you protect your SMB while your people work remotely? Here are 7 security steps to make your business more secure while everyone’s working from home. 1. Talk to your employees and leadership team about phishing Even before the coronavirus emerged, scammers were sending out 3.4 billion phishing emails every day, per TechRadar and Valimail. Now, scammers are targeting remote workers with COVID-19 related: scams designed to steal their login credentials to Office365, OneDrive and other cloud data storage services “urgent” email impersonations of company leaders requesting that employees transfer funds, pay invoices or make donations online. For real. It’s even happened to us at HostGator.messages that encourage recipients to click on a link or attachment for COVID-19 information, only to download ransomware. To help keep your employees from getting phished, David Johnson, Chief Information Security Officer for HostGator, recommends reminding them to watch for emails containing: Mismatched or misleading informationFake shipping or delivery notificationsFake purchase confirmations and invoicesRequests for personal informationPromises of rewardsCharity or gift card requestsUrgent or threatening language (like “your account will be terminated”) Unexpected emails Encourage (and frequently remind) your employees to  Check the email header to see if the sender’s display name and their email match. Scammers can set up a free email account with any name they choose—even yours—so it’s important to check the address.If the email appears to come from a fellow employee, manager, customer or vendor, verify the sender via voice, text or video chat before you follow “urgent” instructions, especially requests for money. Be cautious about clicking links, opening attachments or putting information into pop-up dialog boxes. When in doubt, don’t.Report suspicious emails to you or your IT person.  2. Protect your website from crashes and takeovers Because ransomware attacks are on the rise, it’s also important to make sure your website has regular backups and continuous malware scans.  Check with your IT person or your web hosting provider to make sure you have automated site backups at least once every 24 hours that include file and database backups. That way if your site goes down, you have a recent version you can bring back up while you sort out the problem. You’ll also want to check to see if your site gets regular scans for malware infections and the kinds of vulnerabilities that could allow attackers to inject malware into your site. The quicker these problems are spotted and removed, the better. HostGator customers can add CodeGuard and SiteLock to their websites for regular backups, scans and protection from bot attacks.  3. Protect company equipment from hacking and theft If your employees are using company-issued computers and mobile devices, make the rules for safe use clear.  Company devices should be used for work only. Many companies mandate that company computers should be used for company work only, not as a personal computer. But the employees may not think about how using company tech for personal tasks and leisure creates risk. All it takes is one wrong click for your company data and logins to be hacked. Company devices should be securely stored when not in use. When company laptops and phones get stolen from people’s cars, the data on them gets stolen, too.  4. Keep everyone’s apps and OS up to date Remember that huge Equifax in 2017 data breach that affected more than 143 million people? They could have prevented that by keeping their software patched and up to date. Instead, they let a known vulnerability in one of their apps sit unpatched for weeks, and hackers exploited it.  Keeping your company’s software updated, and patching vulnerabilities as soon as patches are available, are easy ways to keep hackers from walking right into your system.  The need for real-time updates also applies to company-owned devices that your employees are using at home—and to their personal computers and phones if they’re using those to work remotely.  5. Make employees’ remote connections as secure as possible How your employees access company email, databases and files matters, because unsecure connections are another potential entry point for bad actors. Let’s look at connection methods from least to most secure. Public or free Wi-Fi and computers. This shouldn’t be much of an issue right now, because so many of us are confined to home. But just in case, it’s wise to make clear that no employees should be logging in to work from public Wi-Fi or public computer terminals.  Home networks are safer than public Wi-Fi, if they’re set up correctly. Encourage your employees to make sure their home Wi-Fi network password isn’t easy to guess, and that it isn’t used for any other accounts.  If your workforce is at least moderately tech-savvy, you can encourage them to change the default password on their home router. Often, it’s “admin/admin” which makes them vulnerable to hackers who use search tools to scan IP addresses on the web, find those with default router credentials and hijack them. Read our guide to setting a secure password. Your company’s VPN. If you already have a virtual private network (VPN), make sure it’s up to date and require your remote employees to use it.  Don’t have a VPN? Now’s the time to invest in one. A VPN encrypts the data that moves between your company’s system and your remote workers so there’s no way to steal it en route. Not sure where to begin? TechRadar has business VPN recommendations.  6. Have everyone use the same tech tools Even though everyone’s out of the office, they should all be using the apps and services you’ve selected for your company—or alternatives you approve. When employees start using new apps to do their work without an OK from the company, that’s called “shadow IT.” It can result in “serious security gaps,” according to Cisco, in part because shadow IT increases your attack surface. For example, if your company shares documents, slide decks and spreadsheets through Google Drive, no one should also be using Dropbox or OneDrive to share company data. If you’re using Slack for work conversations, employees should stick to Slack and not break off into Skype or Hangouts groups to work.  7. Encourage good password hygiene Reminding your employees to use strong, unique passwords may not seem like an important security step. After all, we’ve been hearing that advice for years but people still use terrible, insecure passwords like 123123.  But terrible, insecure passwords are an easy way for hackers to get into your employees’ accounts and then into your business. One way to ensure better passwords is to use a password manager service for your SMB. With this kind of tool, you can require strong passwords, require two-factor authentication if you like and schedule required password changes. You can find a guide to SMB password managers at InformationWeek’s Dark Reading.  How can you get your employees to follow these recommendations? Too many recommendations at once may overwhelm your people. A more workable approach is to focus on one security task per workday or per week, depending on everyone’s bandwidth. With a planned approach, you can make your business more secure and give everyone one less thing to worry about. Ready to make your SMB site more secure? Get daily site backups with CodeGuard and regular malware scans with SiteLock.  Find the post on the HostGator Blog

The Bandwidth Alliance Charges Forward with New Partners - Alibaba, Zenlayer, and Cherry Servers

CloudFlare Blog -

We started the Bandwidth Alliance in 2018 with a group of like-minded cloud and networking partners. Our common goal was to help our mutual customers reduce or eliminate data transfer charges, sometimes known as "bandwidth” or “egress” fees, between the cloud and the consumer. By reducing or eliminating these costs, our customers can more easily choose a best of breed set of solutions because they don’t have to worry about data charges from moving workloads between vendors, and thereby becoming locked-in to a single provider for all their needs. Today we’re announcing an important milestone: the addition of Alibaba, Zenlayer, and Cherry Servers to the Bandwidth Alliance, expanding it to a total of 20 partners. These partners offer our customers a wide choice of cloud services and products each suited to different needs. In addition, we are working with our existing partners including Microsoft Azure, Digital Ocean and several others to onboard customers and provide them the benefits of the Bandwidth Alliance. Contact us at if you are interested.Customer savings  Over the past year we have seen several customers take advantage of the Bandwidth Alliance and wanted to highlight two examples. Nodecraft, which allows users an easy way to set up their own game servers, is a perfect example of how the Bandwidth Alliance helped to cut down egress costs. Nodecraft supports games like Minecraft, ARK: Survival Evolved, and Counter-Strike. As Nodecraft’s popularity increased, so did their AWS bill. They were not only being charged for storage they were using, but also for ‘egress’ or data transfer fees out of AWS. They made the decision to move their storage to Backblaze. Now they use Backblaze’s B2 Storage and Cloudflare’s extensive network to deliver content to customers without any egress charges. Read more about their journey here and provides simple and smart solutions for podcasting, including hosting, analytics, and ads. The most important of Pippa’s business is rapid and reliable asset delivery. As Pippa grew, they were pushing millions of large audio files to listeners worldwide. This resulted in significantly increased costs, including excessive data egress fees for retrieving data from a cloud storage service such as AWS S3. DigitalOcean waives egress fees to transfer data to Cloudflare, effectively creating a zero-cost data bridge from DigitalOcean to Cloudflare’s global network. Pippa moved to DigitalOcean storage and Cloudflare’s global cloud security and delivery network. With the combination of lower cost storage and zero egress fees, Pippa saw a 50% savings on their cloud bill.What could your savings be?Nodecraft and Pippa are just two examples of small businesses who are seeing significant cost savings from the Bandwidth Alliance. They both chose the best storage and cloud solution for their use case and a global cloud network by Cloudflare without any taxation of transferring data between these two products. With our newly added partners we expect many more customers to benefit.You may be asking - ‘How much can I save?’ To help you get a sense of the scale of your potential savings, by moving to the Bandwidth Alliance, we have put together a calculator. Fill in your details on egress to figure out how much you could be saving. We hope this is a helpful resource as you evaluate your cloud platform choices and spend.

Why We Started Putting Unpopular Assets in Memory

CloudFlare Blog -

Part of Cloudflare's service is a CDN that makes millions of Internet properties faster and more reliable by caching web assets closer to browsers and end users.We make improvements to our infrastructure to make end-user experiences faster, more secure, and more reliable all the time. Here’s a case study of one such engineering effort where something counterintuitive turned out to be the right approach.Our storage layer, which serves millions of cache hits per second globally, is powered by high IOPS NVMe SSDs.Although SSDs are fast and reliable, cache hit tail latency within our system is dominated by the IO capacity of our SSDs. Moreover, because flash memory chips wear out, a non-negligible portion of our operational cost, including the cost of new devices, shipment, labor and downtime, is spent on replacing dead SSDs.Recently, we developed a technology that reduces our hit tail latency and reduces the wear out of SSDs. This technology is a memory-SSD hybrid storage system that puts unpopular assets in memory.The end result: cache hits from our infrastructure are now faster for all customers.You may have thought that was a typo in my explanation of how the technique works. In fact, a few colleagues thought the same when we proposed this project internally, “I think you meant to say ‘popular assets’ in your document”. Intuitively, we should only put popular assets in memory since memory is even faster than SSDs.You are not wrong. But I’m also correct. This blog post explains why.Putting popular assets in memoryFirst let me explain how we already use memory to speed up the IO of popular assets.Page cacheSince it is so obvious that memory speeds up popular assets, Linux already does much of the hard work for us. This functionality is called the “page cache”. Files in Linux systems are organized into pages internally, hence the name.The page cache uses the system’s available memory to cache reads from and buffer writes to durable storage. During typical operations of a Cloudflare server, all the services and processes themselves do not consume all physical memory. The remaining memory is used by the page cache.Below shows the memory layout of a typical Cloudflare edge server with 256GB of physical memory.We can see the used memory (RSS) on top. RSS memory consumption for all services is about 87.71 GB, with our cache service consuming 4.1 GB. At the bottom we see overall page cache usage. Total size is 128.6 GB, with the cache service using 41.6 GB of it to serve cached web assets.Caching readsPages that are frequently used are cached in memory to speed up reads. Linux uses the least recently used (LRU) algorithm among other techniques to decide what to keep in page cache. In other words, popular assets are already in memory.Buffering writesThe page cache also buffers writes. Writes are not synchronized to disk right away. They are buffered until the page cache decides to do so at a later time, either periodically or due to memory pressure. Therefore, it also reduces repetitive writes to the same file.However, because of the nature of the caching workload, a web asset arriving at our cache system is usually static. Unlike workloads of databases, where a single value can get repeatedly updated, in our caching workload, there are very few repetitive updates to an asset before it is completely cached. Therefore, the page cache does not help in reducing writes to disk since all pages will eventually be written to disks.Smarter than the page cache?Although the Linux page cache works great out of the box, our caching system can still outsmart it since our system knows more about the context of the data being accessed, such as the content type, access frequency and the time-to-live value.Instead of improving caching popular assets, which the page cache already does, in the next section, we will focus on something the page cache cannot do well: putting unpopular assets in memory.Putting unpopular assets in memoryMotivation: Writes are badWorn out SSDs are solely caused by writes, not reads. As mentioned above, it is costly to replace SSDs. Since Cloudflare operates data centers in 200 cities across the world, the cost of shipment and replacement can be a significant proportion of the cost of the SSD itself.Moreover, write operations on SSDs slow down read operations. This is because SSD writes require program/erase (P/E) operations issued to the storage chips. These operations block reads to the same chips. In short, the more writes, the slower the reads. Since the page cache already cached popular assets, this effect has the highest impact on our cache hit tail latency. Previously we talked about how we dramatically reduced the impact of such latency. In the meantime, directly reducing the latency itself will also be very helpful.Motivation: Many assets are “one-hit-wonders”One-hit-wonders refer to cached assets that are never accessed; they get accessed once and then cached yet never read again. In addition, one-hit-wonders also include cached assets that are evicted before they are used once due to their lack of popularity. The performance of websites is not harmed even if we don’t cache such assets in the first place.To quantify how many of our assets are one-hit-wonders and to test the hypothesis that reducing disk writes can speed up disk reads, we conducted a simple experiment. The experiment: using a representative sample of traffic, we modified our caching logic to only cache an asset the second time our server encountered it. The red line indicates when the experiment started. The green line represents the experimental group and the yellow line represents the control group.The result: disk writes per second were reduced by roughly half and corresponding disk hit tail latency was reduced by approximately five percent.This experiment demonstrated that if we don’t cache one-hit-wonders, our disks last longer and our cache hits are faster.One more benefit of not caching one-hit-wonders, which was not immediately obvious from the experiment, is that, it increases the effective capacity of cache because of the reduced competing pressure from the removal of the unpopular assets. This in turn increases the cache hit ratio and cache retention.The next question: how can we replicate these results, at scale, in production, without impacting customer experience negatively?Potential implementation approachesRemember but don’t cacheOne smart way to eliminate one-hit-wonders from cache is to not cache an asset during its first few misses but to remember its appearances. When the cache system encounters the same asset repeatedly over a certain number of times, the system will start to cache the asset. This is basically what we did in the experiment above.In order to remember the appearances, beside hash tables, many memory efficient data structures, such as Bloom filter, Counting Bloom filter and Count-min sketch, can be used with their specific trade-offs.Such an approach solves one problem but introduces a new one: every asset is missed at least twice before it is cached. Because the number of cache misses is amplified compared to what we do today, this approach multiplies both the cost of bandwidth and server utilization for our customers. This is not an acceptable tradeoff in our eyes.Transient cache in memoryA better idea we came up with: put every asset we want to cache to disk in memory first. The memory is called a “transient cache”. Assets in transient cache are promoted to the permanent cache, backed by SSDs, after they are accessed a certain number of times, indicating they are popular enough to be stored persistently. If they are not promoted, they are eventually evicted from transient cache because of lack of popularity.This design makes sure that disk writes are only spent on assets that are popular, while ensuring every asset is only “missed” once.The transient cache system on the real world InternetWe implemented this idea and deployed it to our production caching systems. Here are some learnings and data points we would like to share.Trade-offsOur transient cache technology does not pull a performance improvement rabbit out of a hat. It achieves improved performance for our customers by consuming other resources in our system. These resource consumption trade-offs must be carefully considered when deploying to systems at the scale at which Cloudflare operates.Transient cache memory footprintThe amount of memory allocated for our transient cache matters. Cache size directly dictates how long a given asset will live in cache before it is evicted. If the transient cache is too small, new assets will evict old assets before the old assets receive hits that promote them to disk. From a customer's perspective, cache retention being too short is unacceptable because it leads to more misses, commensurate higher cost, and worse eyeball performance.Competition with page cacheAs mentioned before, the page cache uses the system’s available memory. The more memory we allocate to the transient cache for unpopular assets, the less memory we have for popular assets in the page cache. Finding the sweet spot for the trade-off between these two depends on traffic volume, usage patterns, and the specific hardware configuration our software is running on.Competition with process memory usageAnother competitor to transient cache is regular memory used by our services and processes. Unlike the page cache, which is used opportunistically, process memory usage is a hard requirement for the operation of our software. An additional wrinkle is that some of our edge services increase their total RSS memory consumption when performing a “zero downtime upgrade”, which runs both the old and new version of the service in parallel for a short period of time. From our experience, if there is not enough physical memory to do so, the system’s overall performance will degrade to an unacceptable level due to increased IO pressure from reduced page cache space.In productionGiven the considerations above, we enabled this technology conservatively to start. The new hybrid storage system is used only by our newer generations of servers that have more physical memory than older generations. Additionally, the system is only used on a subset of assets. By tuning the size of the transient cache and what percentage of requests use it at all, we are able to explore the sweet spots for the trade-offs between performance, cache retention, and memory resource consumption.The chart below shows the IO usage of an enabled cohort before and after (red line) enabling transient cache.We can see that disk write (in bytes per second) was reduced by 25% at peak and 20% off peak. Although it is too early to tell, the life-span of those SSDs should be extended proportionally.More importantly for our customers: our CDN cache hit tail latency is measurably decreased!Future workWe just made the first step towards a smart memory-SSD hybrid storage system. There is still a lot to be done.Broader deploymentCurrently, we only apply this technology to specific hardware generations and a portion of traffic. When we decommission our older generations of servers and replace them with newer generations with more physical memory, we will be able to apply the transient cache to a larger portion of traffic. Per our preliminary experiments, in certain data centers, applying the transient cache to all traffic is able to reduce disk writes up to 70% and decrease end user visible tail latency by up to 20% at peak hours.Smarter promotion algorithmsOur implemented promotion strategy for moving assets from transient cache to durable storage is based on a simple heuristic: the number of hits. Other information can be used to make a better promotion decision. For example, the TTL (time to live) of an asset can be taken into account as well. One such strategy is to refuse to promote an asset if the TTL of it has only a few seconds left, which will further reduce unnecessary disk writes.Another aspect of the algorithms is demotion. We can actively or passively (during eviction) demote assets from persistent cache to transient cache based on the criteria mentioned above. The demotion itself does not directly reduce writes to persistent cache but it could help cache hit ratio and cache retention if done smartly.Transient cache on other types of storageWe choose memory to host the transient cache because its wear out cost is low (none). This is also the case for HDDs. It is possible to build a hybrid storage system across memory, SSDs and HDDs to find the right balance between their costs and performance characteristics.ConclusionBy putting unpopular assets in memory, we are able to trade-off between memory usage, tail hit latency and SSD lifetimes. With our current configuration, we are able to extend SSD life while reducing tail hit latency, at the expense of available system memory. Transient cache also opens possibilities for future heuristic and storage hierarchy improvements.Whether the techniques described in this post benefit your system depends on the workload and the hardware resource constraints of your system. Hopefully us sharing this counterintuitive idea can inspire other novel ways of building faster and more efficient systems.And finally, if doing systems work like this sounds interesting to you, come work at Cloudflare!

Marketing in Times of Uncertainty: Tips From Top Marketing Pros

Social Media Examiner -

Are you unsure how to approach your marketing during a crisis? Looking for wisdom from well-known marketers? To help you make wise decisions during trying times, we tapped the minds of top marketers to answer these questions: #1: Your Business Must Sell to Survive: Daniel Harmon The Question: Should I continue to promote my business […] The post Marketing in Times of Uncertainty: Tips From Top Marketing Pros appeared first on Social Media Marketing | Social Media Examiner.

Creating Your First WordPress Plugin

InMotion Hosting Blog -

There are many reasons why you may, at some point, want to build your own WordPress plugin. The WordPress core functionality gives you all the necessary features of a content management system (CMS). These include a front end process to generate pages and a back end to manage the content you create. There are also functions to protect the site by locking the back end, so only registered users may access the administrative features. Continue reading Creating Your First WordPress Plugin at InMotion Hosting Blog.


Recommended Content

Subscribe to Complete Hosting Guide aggregator