Distributed Shared Memory: Central Manager Algorithm

A Central Manager is similar to a monitor. It keeps track of the owners and copysets of all the pages in shared distributed memory in a table called Info. Each node must send non-local access requests through the Central Manager. The Central Manager sends the request to the page owner and awaits confirmation of the transfer before releasing the page lock on Info.

A node becomes owner when it requests write access to a page. When a read copy of the page is requested, the owner's access is downgraded to read. If the owner wishes to write to that page again, it must request write access from the manager, who will invalidate all the read copies.

The copyset eliminates the need to broadcast page invalidations to all the nodes. Only those nodes having a read copy of the page need to be notified.

The Fault Handlers handle page requests from local processes. The Servers handle page requests from other nodes.

Access costs are constant with this algorithm. Each request on a non-manager node requires four messages. Each write request has the additional cost of sending invalidation messages. This algorithm can result in a bottleneck at the Central Manager node as the number of processors or page faults increases. The Dynamic Distributed Manager scheme was developed to address this problem.

Data structures

Data structure at each processor: PTable
    one entry per page
    two fields per entry:
        access - type of access this processor has to this page
                 (read/write/nil)
        lock - used to lock access to page for syncronization

Data structure at central manager processor: Info (table)  
    one entry per page
    three fields per entry
      owner - processor that owns page 
              (processor with most recent write access)
      copyset - list of processors that have copies of page
      lock - used to lock access to page for syncronization

Locking Mechanism

Both PTable and Info have locks for each page. This sychronizes local and non-local page requests. The PTable entry locks the page for processor access so that the fault handlers and the read/write servers do not handle requests for the same page at the same time. The Info lock prevents race conditions for the information in the table.

When two processes on the same processor request the same page, the locking mechanism prevents the processor from sending two requests to the owner of the page. The second local request will be blocked on the PTable lock and when the lock is released, the page will already reside on the processor.

If the manager receives write requests for the same page from two different processors, it will change ownership of the page to the first requestor and forward the second request to the new owner, possibly before the first requestor has finished aquiring ownership. The PTable lock at the first requestor node will cause the second request to wait until the node has ownership of the page.

Central Manager Algorithm

Read Fault Handler
    Lock (PTable[page].lock) ;
    IF I am manager THEN BEGIN
        Lock (Info[page].lock) ;
        Info[page].copyset := 
                   Info[page].copyset U {ManagerNode} ;
        receive page from Info[page].owner ;
        Unlock (Info[page].lock) ;
        END ;
    ELSE BEGIN
        ask manager for read access to page & a copy of page ;
        receive page ;
        send confirmation to manager ;
        END ;
    PTable[page].access := read ;
    Unlock (PTable[page].lock) ;
Read Server
    Lock (PTable[page].lock) ;
    IF I am owner THEN BEGIN
        PTable[page].access := read ;
        send copy of page
        END ;
    Unlock (PTable[page].lock) ;

    IF I am manager THEN BEGIN
        Lock (Info[page].lock) ;
        Info[page].copyset := 
                   Info[page].copyset U {RequestNode} ;
        ask Info[page].owner to send copy of page to RequestNode ;
        receive confirmation from RequestNode ;
        Unlock (Info[page].lock) ;
        END ;
Write Fault Handler
    Lock (PTable[page].lock) ;
    IF I am manager THEN BEGIN
        Lock (Info[page].lock) ;
        Invalidate (page, Info[page].copyset) ;
        Info[page].copyset := {} ;
        Unlock (Info[page].lock) ;
        END ;
    ELSE BEGIN
        ask manager for write access to page ;
        receive page ;
        send confirmation to manager ;
        END ;
    PTable[page].access := write ;
    Unlock (PTable[page].lock) ;
Write Server
    Lock (PTable[page].lock) ;
    IF I am owner THEN BEGIN
        send copy of page
        PTable[page].access := nil ;
        END ;
    Unlock (PTable[page].lock) ;

    IF I am manager THEN BEGIN
        Lock (Info[page].lock) ;
        Invalidate (page, Info[page].copyset) ;
        Info[page].copyset := {} ;
        ask Info[page].owner to send page to RequestNode ;
        receive confirmation from RequestNode ;
        Unlock (Info[page].lock) ;
        END ;

You will probably notice that the servers need a test added to the code following "IF I am manager" to prevent the manager from asking for a page that it owns.

    IF I am not owner THEN BEGIN
        ask Info[page].owner to send page to RequestNode ;
        END ;

Diagram of the messages passed to service a read page fault


Reducing Message Traffic

The confirmation message from the requesting node to the Central Manager can be eliminated if the owner of the page keeps the copyset. Each node adds a field to PTable for the copy set, which is valid only if the node is the owner, and sends the invalidate messages whenever it requests write access. The Central Manager node replaces Info with an Owner table that does not contain copysets and does not require locks. The Central Manager forwards requests for pages to the owner and changes the owner in the Owner table if the request is for write access. This change eliminates one send and one receive per page fault on all processors and can help reduce the bottleneck problem at the Central Manager node.


QUESTIONS

  1. What is the major drawback to the central manager algorithm?
  2. What is the purpose of the PTable page lock?
  3. Why does the Central Manager have two locks per page?
  4. Describe a situation where the Central Manager algorithm would provide better performance than the Dynamic Distributed Manager algorithm.
  5. Which algorithm (Central Manager or Dynamic Distributed Manager) would you choose for a real time system. Why?
  6. Why do the fault handlers have mutually exclusive routines for manager and owner?
  7. The manager servers unlock the PTable page lock before making changes to the Info table. Can you describe a situation where this could lead to inconsistency?

REFERENCES

  1. Singhal, M., and N.G. Shivaratri (1994), Advanced Concepts in Operating Systems, McGraw Hill, Inc., New York, NY, pp. 238-240, 248-252.
  2. Li, K., and P. Hudak, "Memory Coherence in Shared Virtual Memory Systems," ACM Transactions on Computer Systems, vol. 7, no. 4, November 1989, pp. 321-359.

By Karen Bowen bowen@cs.vt.edu