Perf to the detail of stack frame

Use perf to profile the redis daemon

The report shows:

I could only see that function “ziplistFind” is very hot in redis but I can’t see how the code routine reaches the “ziplistFind”. By searching around I find this article. So I use perf as:

The report indicates more details in program now:

LZ4 is faster, but not better

I need to compress small size of data in my project without too heavy side-effect on performance. Many people recommend LZ4 for me since it is almost the fastest compression algorithm at present.

Compression ratio is not the most important element here, but it still need to be evaluated. So I create a 4KB file contains normal text (from a README file from a open source software) and compress it with LZ4 and GZIP.

LZ4 GZIP -1
Compression Ratio 1.57 2.24

Hum, Looks like the compression ratio of LZ4 is not bad. But when I run test with this special content (changed from here):

the result became interesting:

LZ4 GZIP -1
Compression Ratio 0.99 2.25

GZIP could compress this content unexceptionally, but LZ4 don’t. So, LZ4 can’t compress some special content (like, numbers), although it is faster than any other compression algorithm.
Somebody may ask: why don’t you use special algorithm to compress content of numbers? The answer is: in real situation, we could not know the type of our small data segments.

Import data to Redis from JSON format

Redis use rdb file to make data persistent. Yesterday, I used the redis-rdb-tools for dumping data from rdb file to JSON format. After that, I write scala code to read data from JSON file and put it into Redis.

Firstly, I found out that the JSON file is almost 50% bigger than rdb file. After checking the whole JSON file, I make sure that the root cause is not the redundant symbols in JSON such as braces and brackets but the “Unicode transformation” in JSON format, especially the “\u0001” for ASCII of “0x01”. Therefore I write code to replace it:

Then the size of JSON file became normal.

There was still another problem. To read the JSON file line by line, I use code from http://naildrivin5.com/blog/2010/01/26/reading-a-file-in-scala-ruby-java.html:

But this code will load all data from file and then run “foreach”. As my file is bigger than 100GB, it will cost too much time and RAM ….
The best way is to use IOStream in java:

This is exactly read file “line by line”.

From scala Array[String] / Seq[String] to java varargs

While testing performance of redis these days, I need to use mset() interface of jedis (a java version redis client). But the prototype of mset() in jedis is:

Firstly I write my scala code like:

But it report compiling errors:

After searching many documents about scala/java on google, I finally find the answer: http://docs.scala-lang.org/style/types.html. So, let’s write code this way:

Then Array[String] of scala changes to varargs in java now. It also viable for Seq[String].

Books I read in year 2015

      No Comments on Books I read in year 2015
Illustrated Network Hardware Famous Cases for DataCenter Mysql Internal: Innodb Engine The Martian Antifragile

The first book is about network hardware, like router, switcher. As a coder, I usually use servers on cloud, therefore haven’t see the real high performance routers (I have sought bare server, 1Gb switcher). This book open my eyes.

The second book is about how to build Datacenter. It’s really a work for architecture, not IT guys.

About two years ago, I worked with Mysql team in my company as a kernel developer. We have used PCIE-card of NAND and flashcache as our solution for Mysql to process hight throughput pressure. But util this year, I have read over the architecture of InnoDB Engine which is the most powerful and effective engine in Mysql. Actually, it’s not so difficult to have a overview of the InnoDB Engine in a book. But, it is still very hard to understand the code of it 🙂

I haven’t go to cinema to watch “The Martian” because I have read it in my Kindle on my commute everyday. It is really a sci-fi story for Geeks who like do research on Computer,Chemistry,Physics,etc. The only question I want to ask the author is:” How could you invent so much troubles on Mars to torture Mark Watney?”

Performance test for unikernels (Rumpkernel and OSv)

Unikernels are specialised, single-address-space machine images constructed by using library operating systems. The concept of Unikernel is very old (Since 1980s in embeded system), but become more and more popular in this cloud computing age for its portability and security.
In recent days, I tested two famous unikernel production: Rumpkernel and OSv by running redis in them.

1. Run redis in Rumpkernel (KVM)
Firstly, build rumpkernel and its environment as “https://github.com/rumpkernel/wiki/wiki/Tutorial%3A-Serve-a-static-website-as-a-Unikernel”, then

2. Run redis in OSv (KVM)
Firsty, build OSv by the tutorial of “https://github.com/cloudius-systems/osv/”, and the virbr0 network (as qemu/kvm usually do), then

3. Run redis on host (centos 7 on bare hardware)

4. Use benchmark tool to test it
I choose memtier_benchmark as the benchmark tool.

5. The test result


Request per second
unikernels
unikernels


Latency (unit for Y axle: micro second)
unikernels
unikernels

Looks the performance of OSv is better than Rumpkernel. But still, they all much slower than running on bare hardware. The bottle-neck in this test case is network, so may be we should find a way to bypass tap or bridge.

How to disable the creation of the rescue initramfs on CentOS 7

I was trying to install new linux kernel (4.4-rc5) in my CentOS 7 server. But when I run “sudo make install” it report

The initramfs-0-rescue-XXX file occupied too much space in boot device. Then I found this article. But after adding dracut_rescue_image=”no” into /etc/dracut.conf, the problem was still exist.
Finally, I use

Therefore, the worked configuration item for dracut is in /usr/lib/dracut/dracut.conf.d/02-rescue.conf instead of /etc/dracut.conf on centos 7. The final solution is

“kmem_cache_create: duplicate cache XXX”

In my kernel module, firstly I wrote:

In centos 7, this module works fine, but after port to centos 6, this kernel module reports:

The key to this problme is in the implementation of kmem_cache_create() in 2.6.32 linux kernel (for centos 6):

After creating a new pool, it only point to ‘name’, not strdup() a new one. But the ‘name’ in my kernel module is a temporary variable (in stack), so it considers the name is “duplicated”.
The correct code should like:

But why the old kernel module did not report error in centos 7? Because in centos 7 the default memory allocator is SLUB, and in centos 6 it is SLAB. They have totally different implementation.

“Cache flush bypassed!” from fio

fio is a effective tool to test IO-performance of a block device (also file system).

Today, my colleague tell me fio has report “Cache flush bypassed!” which means all IO have bypass the device cache. But I can’t agree because the cache of a RAID card usually could only be change by specific tool (such as MegaCLi), but not a test tool.

By review the code of fio:

and the implementation for blockdev_invalidate_cache() is:

Therefore, the “Cache flush bypassed!” is not mean all IO will bypass the buffer of device, but actually means: “fio can’t flush the cache of device, so let’s ignore it”.
If you want to disable the DRAM cache on the RAID card, the correct way is set cache policy of RAID card to “Write Through”: