SH-4492 Create a useful README for llcorehttp.

Tue, 17 Sep 2013 21:55:44 +0000

author
Monty Brandenberg <monty@lindenlab.com>
date
Tue, 17 Sep 2013 21:55:44 +0000
changeset 40724
d3cca5ca0fbe
parent 40723
06b01c301672
child 40725
3053bf46de4b

SH-4492 Create a useful README for llcorehttp.
First edit complete. Use the library in 15 minutes. Describe the
code. Refinements to the initial try. Describe that code. Still
to do: more refinements, how to choose a policy class, FAQ.

indra/llcorehttp/README.Linden file | annotate | diff | revisions
     1.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     1.2 +++ b/indra/llcorehttp/README.Linden	Tue Sep 17 21:55:44 2013 +0000
     1.3 @@ -0,0 +1,467 @@
     1.4 +
     1.5 +
     1.6 +
     1.7 +1.  HTTP fetching in 15 Minutes
     1.8 +
     1.9 +    Let's start with a trivial working example.  You'll need a throwaway
    1.10 +    build of the viewer.  And we'll use indra/newview/llappviewer.cpp as
    1.11 +    our host.
    1.12 +
    1.13 +    Add some needed headers:
    1.14 +
    1.15 +
    1.16 +        #include "httpcommon.h"
    1.17 +        #include "httprequest.h"
    1.18 +        #include "httphandler.h"
    1.19 +
    1.20 +
    1.21 +    You'll need to derive a class from HttpHandler (not HttpHandle).
    1.22 +    This is used to deliver notifications of HTTP completion to your
    1.23 +    code.  Place it near the top, before LLDeferredTaskList, say:
    1.24 +
    1.25 +
    1.26 +        class MyHandler : public LLCore::HttpHandler
    1.27 +        {
    1.28 +        public:
    1.29 +            MyHandler()
    1.30 +            : LLCore::HttpHandler()
    1.31 +            {}
    1.32 +
    1.33 +            virtual void onCompleted(LLCore::HttpHandle /* handle */,
    1.34 +                                     LLCore::HttpResponse * /* response */)
    1.35 +            {
    1.36 +                LL_INFOS("Hack") << "It is happening again." << LL_ENDL;
    1.37 +
    1.38 +                delete this;    // Last statement
    1.39 +            }
    1.40 +        };
    1.41 +
    1.42 +
    1.43 +    Add some statics up there as well:
    1.44 +
    1.45 +
    1.46 +        // Our request object.  Allocate during initialiation.
    1.47 +        static LLCore::HttpRequest * my_request(NULL);
    1.48 +
    1.49 +        // The policy class for HTTP traffic.
    1.50 +        // Use HttpRequest::DEFAULT_POLICY_ID, but DO NOT SHIP WITH THIS VALUE!!
    1.51 +        static LLCore::HttpRequest::policy_t my_policy(LLCore::HttpRequest::DEFAULT_POLICY_ID);
    1.52 +
    1.53 +        // Priority for HTTP requests.  Use 0U.
    1.54 +        static LLCore::HttpRequest::priority_t my_priority(0U);
    1.55 +
    1.56 +
    1.57 +    In LLAppViewer::init() after mAppCoreHttp.init(), create a request object:
    1.58 +
    1.59 +
    1.60 +        my_request = new LLCore::HttpRequest();
    1.61 +
    1.62 +
    1.63 +    In LLAppViewer::mainLoop(), just before entering the while loop,
    1.64 +    we'll kick off one HTTP request:
    1.65 +
    1.66 +
    1.67 +        // Construct a handler object (we'll use the heap this time):
    1.68 +        MyHandler * my_handler = new MyHandler;
    1.69 +
    1.70 +        // Issue a GET request to 'http://www.example.com/' kicking off
    1.71 +        // all the I/O, retry logic, etc.
    1.72 +        LLCore::HttpHandle handle;
    1.73 +        handle = my_request->requestGet(my_policy,
    1.74 +                                        my_priority,
    1.75 +                                        "http://www.example.com/",
    1.76 +                                        NULL,
    1.77 +                                        NULL,
    1.78 +                                        my_handler);
    1.79 +        if (LLCORE_HTTP_HANDLE_INVALID == handle)
    1.80 +        {
    1.81 +            LL_WARNS("Hack") << "Failed to launch HTTP request.  Try again."
    1.82 +                             << LL_ENDL;
    1.83 +        }
    1.84 +
    1.85 +
    1.86 +    Finally, arrange to periodically call update() on the request object
    1.87 +    to find out when the request completes.  This will be done by
    1.88 +    calling the onCompleted() method with status information and
    1.89 +    response data from the HTTP operation.  Add this to the
    1.90 +    LLAppViewer::idle() method after the ping:
    1.91 +
    1.92 +
    1.93 +        my_request->update(0);
    1.94 +
    1.95 +
    1.96 +    That's it.  Build it, run it and watch the log file.  You should get
    1.97 +    the "It is happening again." message indicating that the HTTP
    1.98 +    operation completed in some manner.
    1.99 +
   1.100 +
   1.101 +2.  What Does All That Mean
   1.102 +
   1.103 +    MyHandler/HttpHandler.  This class replaces the Responder-style in
   1.104 +    legacy code.  One method is currently defined.  It is used for all
   1.105 +    request completions, successful or failed:
   1.106 +
   1.107 +
   1.108 +        void onCompleted(LLCore::HttpHandle /* handle */,
   1.109 +                         LLCore::HttpResponse * /* response */);
   1.110 +
   1.111 +
   1.112 +    The onCompleted() method is invoked as a callback during calls to
   1.113 +    HttpRequest::update().  All I/O is completed asynchronously in
   1.114 +    another thread.  But notifications are polled by calling update()
   1.115 +    and invoking a handler for completed requests.
   1.116 +
   1.117 +    In this example, the invocation also deletes the handler (which is
   1.118 +    never referenced by the llcorehttp code again).  But other
   1.119 +    allocation models are possible including handlers shared by many
   1.120 +    requests, stack-based handlers and handlers mixed in with other,
   1.121 +    unrelated classes.
   1.122 +
   1.123 +    LLCore::HttpRequest().  Instances of this class are used to request
   1.124 +    all major functions of the library.  Initialization, starting
   1.125 +    requests, delivering final notification of completion and various
   1.126 +    utility operations are all done via instances.  There is one very
   1.127 +    important rule for instances:
   1.128 +
   1.129 +        Request objects may NOT be shared between threads.
   1.130 +
   1.131 +    my_priority.  The APIs support the idea of priority ordering of
   1.132 +    requests but it hasn't been implemented and the hope is that this
   1.133 +    will become useless and removed from the interface.  Use 0U except
   1.134 +    as noted.
   1.135 +
   1.136 +    my_policy.  This is an important one.  This library attempts to
   1.137 +    manage TCP connection usage more rigorously than in the past.  This
   1.138 +    is done by issuing requests to a queue that has various settable
   1.139 +    properties.  These establish connection usage for the queue as well
   1.140 +    as how queues compete with one another.  (This is patterned after
   1.141 +    class-based queueing used in various networking stacks.)  Several
   1.142 +    classes are pre-defined.  Deciding when to use an existing class and
   1.143 +    when to create a new one will determine what kind of experience
   1.144 +    users have.  We'll pick up this question in detail below.
   1.145 +
   1.146 +    requestGet().  Issues an ordinary HTTP GET request to a given URL
   1.147 +    and associating the request with a policy class, a priority and an
   1.148 +    response handler.  Two additional arguments, not used here, allow
   1.149 +    for additional headers on the request and for per-request options.
   1.150 +    If successful, the call returns a handle whose value is other than
   1.151 +    LLCORE_HTTP_HANDLE_INVALID.  The HTTP operation is then performed
   1.152 +    asynchronously by another thread without any additional work by the
   1.153 +    caller.  If the handle returned is invalid, you can get the status
   1.154 +    code by calling my_request->getStatus().
   1.155 +
   1.156 +    update().  To get notification that the request has completed, a
   1.157 +    call to update() will invoke onCompleted() methods.
   1.158 +
   1.159 +
   1.160 +3.  Refinements, Necessary and Otherwise
   1.161 +
   1.162 +    MyHandler::onCompleted().  You'll want to do something useful with
   1.163 +    your response.  Distinguish errors from successes and getting the
   1.164 +    response body back in some form.
   1.165 +
   1.166 +    Add a new header:
   1.167 +
   1.168 +
   1.169 +        #include "bufferarray.h"
   1.170 +
   1.171 +
   1.172 +    Replace the existing MyHandler::onCompleted() definition with:
   1.173 +
   1.174 +
   1.175 +        virtual void onCompleted(LLCore::HttpHandle /* handle */,
   1.176 +                                 LLCore::HttpResponse * response)
   1.177 +        {
   1.178 +            LLCore::HttpStatus status = response->getStatus();
   1.179 +            if (status)
   1.180 +            {
   1.181 +                // Successful request.  Try to fetch the data
   1.182 +                LLCore::BufferArray * data = response->getBody();
   1.183 +
   1.184 +                if (data && data->size())
   1.185 +                {
   1.186 +                    // There's some data.  A BufferArray is a linked list
   1.187 +                    // of buckets.  We'll create a linear buffer and copy
   1.188 +                    // it into it.
   1.189 +                    size_t data_len = data->size();
   1.190 +                    char * data_blob = new char [data_len + 1];
   1.191 +                    data->read(0, data_blob, data_len);
   1.192 +                    data_blob[data_len] = '\0';
   1.193 +
   1.194 +                    // Process the data now in NUL-terminated string.
   1.195 +                    // Needs more scrubbing but this will do.
   1.196 +                    LL_INFOS("Hack") << "Received:  " << data_blob << LL_ENDL;
   1.197 +
   1.198 +                    // Free the temporary data
   1.199 +                    delete [] data_blob;
   1.200 +                }
   1.201 +            }
   1.202 +            else
   1.203 +            {
   1.204 +                // Something went wrong.  Translate the status to
   1.205 +                // a meaningful message.
   1.206 +                LL_WARNS("Hack") << "HTTP GET failed.  Status:  "
   1.207 +                                 << status.toTerseString()
   1.208 +                                 << ", Reason:  " << status.toString()
   1.209 +                                 << LL_ENDL;
   1.210 +            }           
   1.211 +
   1.212 +            delete this;    // Last statement
   1.213 +        }
   1.214 +
   1.215 +
   1.216 +    HttpHeaders.  The header file "httprequest.h" documents the expected
   1.217 +    important headers that will go out with the request.  You can add to
   1.218 +    these by including an HttpHeaders object with the requestGet() call.
   1.219 +    These are typically setup once as part of init rather than
   1.220 +    dynamically created.
   1.221 +
   1.222 +    Add another header:
   1.223 +
   1.224 +
   1.225 +        #include "httpheaders.h"
   1.226 +
   1.227 +
   1.228 +    In LLAppViewer::mainLoop(), add this alongside the allocation of
   1.229 +    my_handler:
   1.230 +
   1.231 +
   1.232 +        // Additional headers for all requests
   1.233 +        LLCore::HttpHeaders * my_headers = new LLCore::HttpHeaders();
   1.234 +        my_headers->append("Accept", "text/html, application/llsd+xml");
   1.235 +
   1.236 +
   1.237 +    HttpOptions.  Options are similar and include a mix of value types.
   1.238 +    One interesting per-request option is the trace setting.  This
   1.239 +    enables various debug-type messages in the log file that show the
   1.240 +    progress of the request through the library.  It takes values from
   1.241 +    zero to three with higher values giving more verbose logging.  We'll
   1.242 +    use '2' and this will also give us a chance to verify that
   1.243 +    HttpHeaders works as expected.
   1.244 +
   1.245 +    Same as above, a new header:
   1.246 +
   1.247 +
   1.248 +        #include "httpoptions.h"
   1.249 +
   1.250 +
   1.251 +    And in LLAppView::mainLoop():
   1.252 +
   1.253 +
   1.254 +        // Special options for requests
   1.255 +        LLCore::HttpOptions * my_options = new LLCore::HttpOptions();
   1.256 +        my_options->setTrace(2);
   1.257 +
   1.258 +
   1.259 +    Now let's put that all together into a more complete requesting
   1.260 +    sequence.  Replace the existing invocation of requestGet() with this
   1.261 +    slightly more elaborate block:
   1.262 +
   1.263 +
   1.264 +        LLCore::HttpHandle handle;
   1.265 +        handle = my_request->requestGet(my_policy,
   1.266 +                                        my_priority,
   1.267 +                                        "http://www.example.com/",
   1.268 +                                        my_options,
   1.269 +                                        my_headers,
   1.270 +                                        my_handler);
   1.271 +        if (LLCORE_HTTP_HANDLE_INVALID == handle)
   1.272 +        {
   1.273 +             LLCore::HttpStatus status = my_request->getStatus();
   1.274 +
   1.275 +             LL_WARNS("Hack") << "Failed to request HTTP GET.  Status:  "
   1.276 +                              << status.toTerseString()
   1.277 +                              << ", Reason:  " << status.toString()
   1.278 +                              << LL_ENDL;
   1.279 +
   1.280 +             delete my_handler;    // No longer needed.
   1.281 +             my_handler = NULL;
   1.282 +        }
   1.283 +
   1.284 +
   1.285 +    Build, run and examine the log file.  You'll get some new data with
   1.286 +    this run.  First, you should get the www.example.com home page
   1.287 +    content:
   1.288 +
   1.289 +
   1.290 +----------------------------------------------------------------------------
   1.291 +2013-09-17T20:26:51Z INFO: MyHandler::onCompleted: Received:  <!doctype html>
   1.292 +<html>
   1.293 +<head>
   1.294 +    <title>Example Domain</title>
   1.295 +
   1.296 +    <meta charset="utf-8" />
   1.297 +    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
   1.298 +    <meta name="viewport" content="width=device-width, initial-scale=1" />
   1.299 +    <style type="text/css">
   1.300 +    body {
   1.301 +        background-color: #f0f0f2;
   1.302 +        margin: 0;
   1.303 +        padding: 0;
   1.304 +        font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
   1.305 +        
   1.306 +    }
   1.307 +    div {
   1.308 +        width: 600px;
   1.309 +        margin: 5em auto;
   1.310 +        padding: 50px;
   1.311 +        background-color: #fff;
   1.312 +        border-radius: 1em;
   1.313 +    }
   1.314 +    a:link, a:visited {
   1.315 +        color: #38488f;
   1.316 +        text-decoration: none;
   1.317 +    }
   1.318 +    @media (max-width: 700px) {
   1.319 +        body {
   1.320 +            background-color: #fff;
   1.321 +        }
   1.322 +        div {
   1.323 +            width: auto;
   1.324 +            margin: 0 auto;
   1.325 +            border-radius: 0;
   1.326 +            padding: 1em;
   1.327 +        }
   1.328 +    }
   1.329 +    </style>    
   1.330 +</head>
   1.331 +
   1.332 +<body>
   1.333 +<div>
   1.334 +    <h1>Example Domain</h1>
   1.335 +    <p>This domain is established to be used for illustrative examples in documents. You may use this
   1.336 +    domain in examples without prior coordination or asking for permission.</p>
   1.337 +    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
   1.338 +</div>
   1.339 +</body>
   1.340 +</html>
   1.341 +----------------------------------------------------------------------------
   1.342 +
   1.343 +
   1.344 +    You'll also get a detailed trace of the HTTP operation itself.  Note
   1.345 +    the HEADEROUT line which shows the additional header added to the
   1.346 +    request.
   1.347 +
   1.348 +
   1.349 +----------------------------------------------------------------------------
   1.350 +HttpService::processRequestQueue: TRACE, FromRequestQueue, Handle:  086D3148
   1.351 +HttpLibcurl::addOp: TRACE, ToActiveQueue, Handle:  086D3148, Actives:  0, Readies:  0
   1.352 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:  About to connect() to www.example.com port 80 (#0) 
   1.353 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:    Trying 93.184.216.119... 
   1.354 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:  Connected to www.example.com (93.184.216.119) port 80 (#0) 
   1.355 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:  Connected to www.example.com (93.184.216.119) port 80 (#0) 
   1.356 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADEROUT, Data:  GET / HTTP/1.1  Host: www.example.com  Accept-Encoding: deflate, gzip  Connection: keep-alive  Keep-alive: 300  Accept: text/html, application/llsd+xml
   1.357 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  HTTP/1.1 200 OK  
   1.358 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Accept-Ranges: bytes  
   1.359 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Cache-Control: max-age=604800  
   1.360 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Content-Type: text/html  
   1.361 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Date: Tue, 17 Sep 2013 20:26:56 GMT  
   1.362 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Etag: "3012602696"  
   1.363 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Expires: Tue, 24 Sep 2013 20:26:56 GMT  
   1.364 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT  
   1.365 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Server: ECS (ewr/1590)  
   1.366 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  X-Cache: HIT  
   1.367 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  x-ec-custom-error: 1  
   1.368 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Content-Length: 1270  
   1.369 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:    
   1.370 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  DATAIN, Data:  256 Bytes
   1.371 +HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:  Connection #0 to host www.example.com left intact 
   1.372 +HttpLibcurl::completeRequest: TRACE, RequestComplete, Handle:  086D3148, Status:  Http_200
   1.373 +HttpOperation::addAsReply: TRACE, ToReplyQueue, Handle:  086D3148
   1.374 +----------------------------------------------------------------------------
   1.375 +
   1.376 +
   1.377 +4.  What Does All That Mean, Part 2
   1.378 +
   1.379 +    HttpStatus.  The HttpStatus object encodes errors from libcurl, the
   1.380 +    library itself and HTTP status values.  It does this to avoid
   1.381 +    collapsing all non-HTTP error into a single '499' HTTP status and to
   1.382 +    make errors distinct.
   1.383 +
   1.384 +    To aid programming, the usual bool conversions are available so that
   1.385 +    you can write 'if (status)' and the expected thing will happen
   1.386 +    whether it's an HTTP, libcurl or library error.  There's also
   1.387 +    provision to override the treatment of HTTP errors (making 404 a
   1.388 +    success, say).
   1.389 +
   1.390 +    Share data, don't copy it.  The library was started with the goal of
   1.391 +    avoiding data copies as much as possible.  Instead, read-only data
   1.392 +    sharing across threads with atomic reference counts is used for a
   1.393 +    number of data types.  These currently are:
   1.394 +
   1.395 +        * BufferArray.  Linked list of data blocks/HTTP bodies.
   1.396 +        * HttpHeaders.  Shared headers for both requests and responses.
   1.397 +        * HttpOptions.  Request-only data modifying HTTP behavior.
   1.398 +        * HttpResponse.  HTTP response description given to onCompleted.
   1.399 +
   1.400 +    Using objects of these types requires a few rules:
   1.401 +
   1.402 +        * Constructor always gives a reference to caller.
   1.403 +        * References are dropped with release() not delete.
   1.404 +        * Additional references may be taken out with addRef().
   1.405 +        * Unless otherwise stated, once an object is shared with another
   1.406 +          thread it should be treated as read-only.  There's no
   1.407 +          synchronization on the objects themselves.
   1.408 +
   1.409 +    HttpResponse.  You'll encounter this mainly in onCompleted() methods.
   1.410 +    Commonly-used interfaces on this object:
   1.411 +
   1.412 +        * getStatus() to return the final status of the request.
   1.413 +        * getBody() to retrieve the response body which may be NULL or
   1.414 +          zero-length.
   1.415 +        * getContentType() to return the value of the 'Content-Type'
   1.416 +          header or an empty string if none was sent.
   1.417 +
   1.418 +    This is a reference-counted object so you can call addRef() on it
   1.419 +    and hold onto the response for an arbitrary time.  But you'll
   1.420 +    usually just call a few methods and return from onCompleted() whose
   1.421 +    caller will release the object.
   1.422 +
   1.423 +    BufferArray.  The core data representation for request and response
   1.424 +    bodies.  In HTTP responses, it's fetched with the getBody() method
   1.425 +    and may be NULL or non-NULL but zero length.  All successful data
   1.426 +    handling should check both conditions before attempting to fetch
   1.427 +    data from the object.  Data access model uses simple read/write
   1.428 +    semantics:
   1.429 +
   1.430 +        * append()
   1.431 +        * size()
   1.432 +        * read()
   1.433 +        * write()
   1.434 +
   1.435 +    There is a more sophisticated stream adapter that extends these
   1.436 +    methods and will be covered below.  So, one way to retrieve data
   1.437 +    from a request is as follows:
   1.438 +
   1.439 +
   1.440 +        LLCore::BufferArray * data = response->getBody();
   1.441 +        if (data && data->size())
   1.442 +        {
   1.443 +            size_t data_len = data->size();
   1.444 +            char * data_blob = new char [data_len + 1];
   1.445 +            data->read(0, data_blob, data_len);
   1.446 +
   1.447 +
   1.448 +    HttpOptions and HttpResponse.  Really just simple containers of POD
   1.449 +    and std::string pairs.  But reference counted and the rule about not
   1.450 +    modifying after sharing must be followed.  You'll have the urge to
   1.451 +    change options dynamically at some point.  And you'll try to do that
   1.452 +    by just writing new values to the shared object.  And in tests
   1.453 +    everything will appear to work.  Then you ship and people in the
   1.454 +    real world start hitting read/write races in strings and then crash.
   1.455 +
   1.456 +    HttpHandle.  Uniquely identifies a request and can be used to
   1.457 +    identify it in an onCompleted() method or cancel it if it's still
   1.458 +    queued.  But as soon as a request's onCompleted() invocation
   1.459 +    returns, the handle becomes invalid and may be reused immediately
   1.460 +    for new requests.  Don't hold on to handles after notification.
   1.461 +
   1.462 +
   1.463 +5.  And Still More Refinements
   1.464 +
   1.465 +
   1.466 +6.  Choosing a Policy Class
   1.467 +
   1.468 +
   1.469 +7.  FAQ
   1.470 +

mercurial