Buffer

   Go标准库Buffer是一个可变大小的字节缓冲区,可以用Wirte和Read方法操作它,在Go标准库中,定义了如下关于Buffer的数据结构。

type Buffer struct {    buf       []byte            // contents are the bytes buf[off : len(buf)]    off       int               // read at &buf[off], write at &buf[len(buf)]    runeBytes [utf8.UTFMax]byte // avoid allocation of slice on each WriteByte or Rune    bootstrap [64]byte          // memory to hold first slice; helps small buffers (Printf) avoid allocation.    lastRead  readOp            // last read operation, so that Unread* can work correctly.}// The readOp constants describe the last action performed on// the buffer, so that UnreadRune and UnreadByte can// check for invalid usage.type readOp intconst (    opInvalid  readOp = iota // Non-read operation.    opReadRune               // Read rune.    opRead                   // Any other read operation.)

   如上定义,Buffer存储的数据是在off到len(buf)区域之间,其他区域是没有数据,而且只能从&buf[off]开始读取数据和从&buf[len(buf)]写数据,同时为了避免对内存的多次操作,对于小的缓冲区,Buffer定义了bootstrap来避免多次内存的操作,runeBytes的定义也是如此目的,还有一个表示对Buffer的操作标识符lastRead。

Buffer的常见操作

  1. 初始化Buffer

    func NewBuffer(buf []byte) *Buffer { return &Buffer{buf: buf} }func NewBufferString(s string) *Buffer {    return &Buffer{buf: []byte(s)}}

    方法NewBuffer使用buf作为参数初始化Buffer,Buffer既可以被读也可以被写,如果是读Buffer,buf需填充一定的数据,如果是写,buf需有一定的容量(capacity),当然也可以通过new(Buffer)来初始化Buffer。另外一个方法NewBufferString用一个string来初始化可读Buffer,并用string的内容填充Buffer.

  2. 读写操作

    func (b *Buffer) Read(p []byte) (n int, err error)func (b *Buffer) Next(n int) []bytefunc (b *Buffer) ReadByte() (c byte, err error)func (b *Buffer) ReadRune() (r rune, size int, err error)func (b *Buffer) ReadBytes(delim byte) (line []byte, err error)func (b *Buffer) readSlice(delim byte) (line []byte, err error)func (b *Buffer) ReadString(delim byte) (line string, err error)func (b *Buffer) Write(p []byte) (n int, err error)func (b *Buffer) WriteString(s string) (n int, err error)func (b *Buffer) ReadFrom(r io.Reader) (n int64, err error)func (b *Buffer) WriteTo(w io.Writer) (n int64, err error)func (b *Buffer) WriteByte(c byte) errorfunc (b *Buffer) WriteRune(r rune) (n int, err error)

    下面对Read,ReadRune,ReadBytes方法进行分析,对于方法Read, 其主要做三个步骤:第一,判断Buffer是否为空,如果是,则重置Buffer;第二,复制Buffer的buf的数据到p,并调整off的位置标识Buffer的可读位置;第三,设置读标识符为opRead。

    func (b *Buffer) Read(p []byte) (n int, err error) {    b.lastRead = opInvalid    if b.off >= len(b.buf) {        // Buffer is empty, reset to recover space.        b.Truncate(0)        if len(p) == 0 {            return        }        return 0, io.EOF    }    n = copy(p, b.buf[b.off:])    b.off += n    if n > 0 {        b.lastRead = opRead    }    return}

   方法ReadRune()定义了如何读取Buffer中UTF8编码的rune数据,同样也需三个步骤,第一,判断Buffer是否为空,若是,重置Buffer;第二,设置读操作符为opReadRune;第三,判断可读位置off处的byte是否小于utf8.Runeself,若是,调整off位置并返回。否则,将Buffer的数据解码成rune,调整off位置,返回解码后的rune及大小。

// ReadRune reads and returns the next UTF-8-encoded// Unicode code point from the buffer.// If no bytes are available, the error returned is io.EOF.// If the bytes are an erroneous UTF-8 encoding, it// consumes one byte and returns U+FFFD, 1.func (b *Buffer) ReadRune() (r rune, size int, err error) {    b.lastRead = opInvalid    if b.off >= len(b.buf) {        // Buffer is empty, reset to recover space.        b.Truncate(0)        return 0, 0, io.EOF    }    b.lastRead = opReadRune    c := b.buf[b.off]    if c < utf8.RuneSelf {        b.off++        return rune(c), 1, nil    }    r, n := utf8.DecodeRune(b.buf[b.off:])    b.off += n    return r, n, nil}

 方法ReadBytes(delim byte)读取Buffer中从off到第一次delim之间的数据,并且包括delim,ReadBytes调用私有方法readSlice来实现,readSlice方法首先查找delim的位置,如果不存在,则返回从off到len(buf)之间的数据,如果存在,则返回off到off+location(delim)+1之间数据,其中加1是为了包括delim,最后设置操作标识符为opRead。

// ReadBytes reads until the first occurrence of delim in the input,// returning a slice containing the data up to and including the delimiter.// If ReadBytes encounters an error before finding a delimiter,// it returns the data read before the error and the error itself (often io.EOF).// ReadBytes returns err != nil if and only if the returned data does not end in// delim.func (b *Buffer) ReadBytes(delim byte) (line []byte, err error) {    slice, err := b.readSlice(delim)    // return a copy of slice. The buffer's backing array may    // be overwritten by later calls.    line = append(line, slice...)    return}// readSlice is like ReadBytes but returns a reference to internal buffer data.func (b *Buffer) readSlice(delim byte) (line []byte, err error) {    i := IndexByte(b.buf[b.off:], delim)    end := b.off + i + 1    if i < 0 {        end = len(b.buf)        err = io.EOF    }    line = b.buf[b.off:end]    b.off = end    b.lastRead = opRead    return line, err}

 同样对相应的Write,WriteRune,ReadFrom, WriteTo写方法进行分析, 对于方法Write,相对Read方法来说,要简单些,主要是扩展Buffer空间,然后将p中的数据复制到Buffer。

// Write appends the contents of p to the buffer, growing the buffer as// needed. The return value n is the length of p; err is always nil. If the// buffer becomes too large, Write will panic with ErrTooLarge.func (b *Buffer) Write(p []byte) (n int, err error) {    b.lastRead = opInvalid    m := b.grow(len(p))    return copy(b.buf[m:], p), nil}

 对于方法WriteRune,首先判断要写的数据rune是否小于utf8.RuneSelf,若是,调用WriteByte将其写入Buffer,若不是,则将要写的数据rune编码成utf8,并调用Write将其写入Buffer。

// WriteRune appends the UTF-8 encoding of Unicode code point r to the// buffer, returning its length and an error, which is always nil but is// included to match bufio.Writer's WriteRune. The buffer is grown as needed;// if it becomes too large, WriteRune will panic with ErrTooLarge.func (b *Buffer) WriteRune(r rune) (n int, err error) {    if r < utf8.RuneSelf {        b.WriteByte(byte(r))        return 1, nil    }    n = utf8.EncodeRune(b.runeBytes[0:], r)    b.Write(b.runeBytes[0:n])    return n, nil}

ReadFrom方法从io.Reader或者实现io.Reader接口的实例中读取所有数据到Buffer,默认情况下最少读取512字节,如果Buffer空间不足512,需增加Buffer空间,该方法返回读取的字节数以及错误信息。从下面可知ReadFrom首先判断Buffer是否为空,若空,则重置Buffer;其次是判断Buffer的free空间是否足够,若小于512且off+free小于512,表示Buffer从0到off之间的空间不足以存放当前Buffer中未读数据的大小,此时设置一临时缓冲区并使其空间Buffer的2倍加上MinRead(512)的空间,将原来Buffer的数据复制到临时缓冲区,然后再把临时缓冲区的数据复制到源Buffer,最后使用io.Reader的Read方法从io.Reader中读取数据,直到遇到io.EOF。

// ReadFrom reads data from r until EOF and appends it to the buffer, growing// the buffer as needed. The return value n is the number of bytes read. Any// error except io.EOF encountered during the read is also returned. If the// buffer becomes too large, ReadFrom will panic with ErrTooLarge.func (b *Buffer) ReadFrom(r io.Reader) (n int64, err error) {    b.lastRead = opInvalid    // If buffer is empty, reset to recover space.    if b.off >= len(b.buf) {        b.Truncate(0)    }    for {        if free := cap(b.buf) - len(b.buf); free < MinRead {            // not enough space at end            newBuf := b.buf            if b.off+free < MinRead {                // not enough space using beginning of buffer;                // double buffer capacity                newBuf = makeSlice(2*cap(b.buf) + MinRead)            }            copy(newBuf, b.buf[b.off:])            b.buf = newBuf[:len(b.buf)-b.off]            b.off = 0        }        m, e := r.Read(b.buf[len(b.buf):cap(b.buf)])        b.buf = b.buf[0 : len(b.buf)+m]        n += int64(m)        if e == io.EOF {            break        }        if e != nil {            return n, e        }    }    return n, nil // err is EOF, so return nil explicitly}
 相对ReadFrom方法,WriteTo方法比较简单,WriteTo将Buffer中的数据写到io.Writer,直到Buffer中没有数据,当Buffer为空时,重置Buffer并返回。
// WriteTo writes data to w until the buffer is drained or an error occurs.// The return value n is the number of bytes written; it always fits into an// int, but it is int64 to match the io.WriterTo interface. Any error// encountered during the write is also returned.func (b *Buffer) WriteTo(w io.Writer) (n int64, err error) {    b.lastRead = opInvalid    if b.off < len(b.buf) {        nBytes := b.Len()        m, e := w.Write(b.buf[b.off:])        if m > nBytes {            panic("bytes.Buffer.WriteTo: invalid Write count")        }        b.off += m        n = int64(m)        if e != nil {            return n, e        }        // all bytes should have been written, by definition of        // Write method in io.Writer        if m != nBytes {            return n, io.ErrShortWrite        }    }    // Buffer is now empty; reset.    b.Truncate(0)    return}

3. 扩展空间和重置  

  Buffer的重置方法Reset()通过调用Truncate(n int)方法来实现清除Buffer的数据,Truncate丢弃除了从off开始的n个未读数据之外的所有数据, 如果n为0,那就重置Buffer。

 

// Truncate discards all but the first n unread bytes from the buffer.// It panics if n is negative or greater than the length of the buffer.func (b *Buffer) Truncate(n int) {    b.lastRead = opInvalid    switch {    case n < 0 || n > b.Len():        panic("bytes.Buffer: truncation out of range")    case n == 0:        // Reuse buffer space.        b.off = 0    }    b.buf = b.buf[0 : b.off+n]}// Reset resets the buffer so it has no content.// b.Reset() is the same as b.Truncate(0).func (b *Buffer) Reset() { b.Truncate(0) }
 在对Buffer进行写数据时,通常需要扩展其空间来使所有的数据都能写入Buffer,Buffer用Grow(n int)方法来实现扩展Buffer空间的功能,该方法调用私有方法grow(n int)。

 

// grow grows the buffer to guarantee space for n more bytes.// It returns the index where bytes should be written.// If the buffer can't grow it will panic with ErrTooLarge.func (b *Buffer) grow(n int) int {    m := b.Len()    // 如果Buffer为空,重置Buffer    if m == 0 && b.off != 0 {        b.Truncate(0)    }    //空间增加n后超过Buffer的容量    if len(b.buf)+n > cap(b.buf) {        //声明一个临时buf        var buf []byte        //Buffer的buf只是被声明,还没有初始化,如果n小于bootstrap的空间,        //直接将boostrap赋值给buf避免内存的操作而增加负载。        if b.buf == nil && n <= len(b.bootstrap) {            buf = b.bootstrap[0:]        //如果满足此条件,滑动b.buf的数据而不是分配一个新的slice空间,然后将b.buf的数据复制给你buf。        } else if m+n <= cap(b.buf)/2 {            copy(b.buf[:], b.buf[b.off:])            buf = b.buf[:m]        } else {            //空间不足,重新分配空间            buf = makeSlice(2*cap(b.buf) + n)            copy(buf, b.buf[b.off:])        }        b.buf = buf        b.off = 0    }    //扩展n的空间,并返回可以写数据的位置    b.buf = b.buf[0 : b.off+m+n]    return b.off + m}// Grow grows the buffer's capacity, if necessary, to guarantee space for// another n bytes. After Grow(n), at least n bytes can be written to the// buffer without another allocation.// If n is negative, Grow will panic.// If the buffer can't grow it will panic with ErrTooLarge.func (b *Buffer) Grow(n int) {    if n < 0 {        panic("bytes.Buffer.Grow: negative count")    }    m := b.grow(n)    b.buf = b.buf[0:m]}