SH-4311 Get highwater limiting of requests into llhttpcore working.

Thu, 27 Jun 2013 20:38:18 -0400

author
Monty Brandenberg <monty@lindenlab.com>
date
Thu, 27 Jun 2013 20:38:18 -0400
changeset 40687
ed725f155ee1
parent 40686
08c1824b4ac3
child 40688
56954bea245d

SH-4311 Get highwater limiting of requests into llhttpcore working.
Fixed the logic and have it covering all five types of requests now
with validation via an assert (when enabled). Should keep things
working smoothly and avoid floods of 503s when in debug modes. Also
started a round of file-level documentation detailing thread usage
and mutex coverage. More to do, more to describe. But the high-
water stuff is functioning correctly.

indra/newview/llmeshrepository.cpp file | annotate | diff | revisions
indra/newview/llmeshrepository.h file | annotate | diff | revisions
     1.1 --- a/indra/newview/llmeshrepository.cpp	Thu Jun 27 13:55:05 2013 -0400
     1.2 +++ b/indra/newview/llmeshrepository.cpp	Thu Jun 27 20:38:18 2013 -0400
     1.3 @@ -76,9 +76,140 @@
     1.4  
     1.5  #include <queue>
     1.6  
     1.7 +
     1.8 +// [ Disclaimer:  this documentation isn't by one of the original authors
     1.9 +//   but by someone coming through later and extracting intent and function.
    1.10 +//   Some of this will be wrong so use judgement. ]
    1.11 +//
    1.12 +// Purpose
    1.13 +//
    1.14 +//   The purpose of this module is to provide access between the viewer
    1.15 +//   and the asset system as regards to mesh objects.
    1.16 +//
    1.17 +//   * High-throughput download of mesh assets from servers while
    1.18 +//     following best industry practices for network profile.
    1.19 +//   * Reliable expensing and upload of new mesh assets.
    1.20 +//   * Recovery and retry from errors when appropriate.
    1.21 +//   * Decomposition of mesh assets for preview and uploads.
    1.22 +//   * And most important:  all of the above without exposing the
    1.23 +//     main thread to stalls due to deep processing or thread
    1.24 +//     locking actions.  In particular, the following operations
    1.25 +//     on LLMeshRepository are very averse to any stalls:
    1.26 +//     * loadMesh
    1.27 +//     * getMeshHeader (For structural details, see:
    1.28 +//       http://wiki.secondlife.com/wiki/Mesh/Mesh_Asset_Format)
    1.29 +//     * notifyLoadedMeshes
    1.30 +//
    1.31 +// Threads
    1.32 +//
    1.33 +//   main     Main rendering thread, very sensitive to locking and other stalls
    1.34 +//   repo     Overseeing worker thread associated with the LLMeshRepoThread class
    1.35 +//   decom    Worker thread for mesh decomposition requests
    1.36 +//   core     HTTP worker thread:  does the work but doesn't intrude here
    1.37 +//   uploadN  0-N temporary mesh upload threads
    1.38 +//
    1.39 +// Mutexes
    1.40 +//
    1.41 +//   LLMeshRepository::mMeshMutex
    1.42 +//   LLMeshRepoThread::mMutex
    1.43 +//   LLMeshRepoThread::mHeaderMutex
    1.44 +//   LLMeshRepoThread::mSignal (LLCondition)
    1.45 +//   LLPhysicsDecomp::mSignal (LLCondition)
    1.46 +//   LLPhysicsDecomp::mMutex
    1.47 +//   LLMeshUploadThread::mMutex
    1.48 +//
    1.49 +// Mutex Order Rules
    1.50 +//
    1.51 +//   1.  LLMeshRepoThread::mMutex before LLMeshRepoThread::mHeaderMutex
    1.52 +//   2.  LLMeshRepository::mMeshMutex before LLMeshRepoThread::mMutex
    1.53 +//   (There are more rules, haven't been extracted.)
    1.54 +//
    1.55 +// Data Member Access/Locking
    1.56 +//
    1.57 +//   Description of how shared access to static and instance data
    1.58 +//   members is performed.  Each member is followed by the name of
    1.59 +//   the mutex, if any, covering the data and then a list of data
    1.60 +//   access models each of which is a triplet of the following form:
    1.61 +//
    1.62 +//     {ro, wo, rw}.{main, repo, any}.{mutex, none}
    1.63 +//     Type of access:  read-only, write-only, read-write.
    1.64 +//     Accessing thread or 'any'
    1.65 +//     Relevant mutex held during access (several may be held) or 'none'
    1.66 +//
    1.67 +//   A careful eye will notice some unsafe operations.  Many of these
    1.68 +//   have an alibi of some form.  Several types of alibi are identified
    1.69 +//   and listed here:
    1.70 +//
    1.71 +//     [0]  No alibi.  Probably unsafe.
    1.72 +//     [1]  Single-writer, self-consistent readers.  Old data must
    1.73 +//          be tolerated by any reader but data will come true eventually.
    1.74 +//     [2]  Like [1] but provides a hint about thread state.  These
    1.75 +//          may be unsafe.
    1.76 +//     [3]  empty() check outside of lock.  Can me made safish when
    1.77 +//          done in double-check lock style.  But this depends on
    1.78 +//          std:: implementation and memory model.
    1.79 +//     [4]  Appears to be covered by a mutex but doesn't need one.
    1.80 +//     [5]  Read of a double-checked lock.
    1.81 +//
    1.82 +//   So, in addition to documentation, take this as a to-do/review
    1.83 +//   list and see if you can improve things.
    1.84 +//
    1.85 +//   LLMeshRepository:
    1.86 +//
    1.87 +//     sBytesReceived
    1.88 +//     sHTTPRequestCount
    1.89 +//     sHTTPRetryCount
    1.90 +//     sLODPending
    1.91 +//     sLODProcessing
    1.92 +//     sCacheBytesRead
    1.93 +//     sCacheBytesWritten
    1.94 +//     mLoadingMeshes                  none            rw.main.none, rw.main.mMeshMutex [4]
    1.95 +//     mSkinMap                        none            rw.main.none
    1.96 +//     mDecompositionMap               none            rw.main.none
    1.97 +//     mPendingRequests                mMeshMutex [4]  rw.main.mMeshMutex
    1.98 +//     mLoadingSkins                   mMeshMutex [4]  rw.main.mMeshMutex
    1.99 +//     mPendingSkinRequests            mMeshMutex [4]  rw.main.mMeshMutex
   1.100 +//     mLoadingDecompositions          mMeshMutex [4]  rw.main.mMeshMutex
   1.101 +//     mPendingDecompositionRequests   mMeshMutex [4]  rw.main.mMeshMutex
   1.102 +//     mLoadingPhysicsShapes           mMeshMutex [4]  rw.main.mMeshMutex
   1.103 +//     mPendingPhysicsShapeRequests    mMeshMutex [4]  rw.main.mMeshMutex
   1.104 +//     mUploads                        none            rw.main.none (upload thread accessing objects)
   1.105 +//     mUploadWaitList                 none            rw.main.none (upload thread accessing objects)
   1.106 +//     mInventoryQ                     mMeshMutex [4]  rw.main.mMeshMutex, ro.main.none [5]
   1.107 +//     mUploadErrorQ                   mMeshMutex      rw.main.mMeshMutex, rw.any.mMeshMutex
   1.108 +//     mGetMeshCapability              none            rw.main.none [0], ro.any.none
   1.109 +//     mGetMesh2Capability             none            rw.main.none [0], ro.any.none
   1.110 +//
   1.111 +//   LLMeshRepoThread:
   1.112 +//
   1.113 +//     sActiveHeaderRequests    mMutex        rw.any.mMutex, ro.repo.none [1]
   1.114 +//     sActiveLODRequests       mMutex        rw.any.mMutex, ro.repo.none [1]
   1.115 +//     sMaxConcurrentRequests   mMutex        wo.main.none, ro.repo.none, ro.main.mMutex
   1.116 +//     mWaiting                 mMutex        rw.repo.none, ro.main.none [2] (race - hint)
   1.117 +//     mMeshHeader              mHeaderMutex  rw.repo.mHeaderMutex, ro.main.mHeaderMutex, ro.main.none [0]
   1.118 +//     mMeshHeaderSize          mHeaderMutex  rw.repo.mHeaderMutex
   1.119 +//     mSkinRequests            none          rw.repo.none, rw.main.none [0]
   1.120 +//     mSkinInfoQ               none          rw.repo.none, rw.main.none [0]
   1.121 +//     mDecompositionRequests   none          rw.repo.none, rw.main.none [0]
   1.122 +//     mPhysicsShapeRequests    none          rw.repo.none, rw.main.none [0]
   1.123 +//     mDecompositionQ          none          rw.repo.none, rw.main.none [0]
   1.124 +//     mHeaderReqQ              mMutex        ro.repo.none [3], rw.repo.mMutex, rw.any.mMutex
   1.125 +//     mLODReqQ                 mMutex        ro.repo.none [3], rw.repo.mMutex, rw.any.mMutex
   1.126 +//     mUnavailableQ            mMutex        rw.repo.none [0], ro.main.none [3], rw.main.mMutex
   1.127 +//     mLoadedQ                 mMutex        rw.repo.mMutex, ro.main.none [3], rw.main.mMutex
   1.128 +//     mPendingLOD              mMutex        rw.repo.mMutex, rw.any.mMutex
   1.129 +//
   1.130 +//   LLPhysicsDecomp:
   1.131 +//    
   1.132 +//     mRequestQ
   1.133 +//     mCurRequest
   1.134 +//     mCompletedQ
   1.135 +//
   1.136 +
   1.137 +
   1.138  LLMeshRepository gMeshRepo;
   1.139  
   1.140 -const S32 MESH_HEADER_SIZE = 4096;						// Important:  assumption is that headers fit in this space
   1.141 +const S32 MESH_HEADER_SIZE = 4096;                      // Important:  assumption is that headers fit in this space
   1.142  const U32 MAX_MESH_REQUESTS_PER_SECOND = 100;
   1.143  const S32 REQUEST_HIGH_WATER_MIN = 32;
   1.144  const S32 REQUEST_HIGH_WATER_MAX = 80;
   1.145 @@ -727,12 +858,21 @@
   1.146  				}
   1.147  			}
   1.148  
   1.149 -			{ //mSkinRequests is protected by mSignal
   1.150 +			// For the final three request lists, if we scan any part of one
   1.151 +			// list, we scan the entire thing.  This gets us through any requests
   1.152 +			// which can be resolved in the cache.  It also keeps the request
   1.153 +			// set somewhat fresher otherwise items at the end of the set
   1.154 +			// order will lose.  Keep to the throttle enforcement and pay
   1.155 +			// attention to the highwater level (enforced in each fetchXXX()
   1.156 +			// method).
   1.157 +			if (! mSkinRequests.empty() && count < MAX_MESH_REQUESTS_PER_SECOND && mHttpRequestSet.size() < sRequestHighWater)
   1.158 +			{
   1.159 +				// *FIXME:  this really does need a lock as do the following ones
   1.160  				std::set<LLUUID> incomplete;
   1.161  				for (std::set<LLUUID>::iterator iter = mSkinRequests.begin(); iter != mSkinRequests.end(); ++iter)
   1.162  				{
   1.163  					LLUUID mesh_id = *iter;
   1.164 -					if (!fetchMeshSkinInfo(mesh_id))
   1.165 +					if (!fetchMeshSkinInfo(mesh_id, count))
   1.166  					{
   1.167  						incomplete.insert(mesh_id);
   1.168  					}
   1.169 @@ -740,12 +880,13 @@
   1.170  				mSkinRequests.swap(incomplete);
   1.171  			}
   1.172  
   1.173 -			{ //mDecompositionRequests is protected by mSignal
   1.174 +			if (! mDecompositionRequests.empty() && count < MAX_MESH_REQUESTS_PER_SECOND && mHttpRequestSet.size() < sRequestHighWater)
   1.175 +			{
   1.176  				std::set<LLUUID> incomplete;
   1.177  				for (std::set<LLUUID>::iterator iter = mDecompositionRequests.begin(); iter != mDecompositionRequests.end(); ++iter)
   1.178  				{
   1.179  					LLUUID mesh_id = *iter;
   1.180 -					if (!fetchMeshDecomposition(mesh_id))
   1.181 +					if (!fetchMeshDecomposition(mesh_id, count))
   1.182  					{
   1.183  						incomplete.insert(mesh_id);
   1.184  					}
   1.185 @@ -753,18 +894,23 @@
   1.186  				mDecompositionRequests.swap(incomplete);
   1.187  			}
   1.188  
   1.189 -			{ //mPhysicsShapeRequests is protected by mSignal
   1.190 +			if (! mPhysicsShapeRequests.empty() && count < MAX_MESH_REQUESTS_PER_SECOND && mHttpRequestSet.size() < sRequestHighWater)
   1.191 +			{
   1.192  				std::set<LLUUID> incomplete;
   1.193  				for (std::set<LLUUID>::iterator iter = mPhysicsShapeRequests.begin(); iter != mPhysicsShapeRequests.end(); ++iter)
   1.194  				{
   1.195  					LLUUID mesh_id = *iter;
   1.196 -					if (!fetchMeshPhysicsShape(mesh_id))
   1.197 +					if (!fetchMeshPhysicsShape(mesh_id, count))
   1.198  					{
   1.199  						incomplete.insert(mesh_id);
   1.200  					}
   1.201  				}
   1.202  				mPhysicsShapeRequests.swap(incomplete);
   1.203  			}
   1.204 +
   1.205 +			// For dev purposes, a dynamic change could make this false
   1.206 +			// and that shouldn't assert.
   1.207 +			// llassert_always(mHttpRequestSet.size() <= sRequestHighWater);
   1.208  		}
   1.209  	}
   1.210  	
   1.211 @@ -927,8 +1073,8 @@
   1.212  }
   1.213  
   1.214  
   1.215 -bool LLMeshRepoThread::fetchMeshSkinInfo(const LLUUID& mesh_id)
   1.216 -{ //protected by mMutex
   1.217 +bool LLMeshRepoThread::fetchMeshSkinInfo(const LLUUID& mesh_id, U32& count)
   1.218 +{
   1.219  	
   1.220  	if (!mHeaderMutex)
   1.221  	{
   1.222 @@ -985,6 +1131,10 @@
   1.223  			}
   1.224  
   1.225  			//reading from VFS failed for whatever reason, fetch from sim
   1.226 +			if (count >= MAX_MESH_REQUESTS_PER_SECOND || mHttpRequestSet.size() >= sRequestHighWater)
   1.227 +			{
   1.228 +				return false;
   1.229 +			}
   1.230  			int cap_version(1);
   1.231  			std::string http_url = constructUrl(mesh_id, &cap_version);
   1.232  			if (!http_url.empty())
   1.233 @@ -1018,8 +1168,8 @@
   1.234  	return ret;
   1.235  }
   1.236  
   1.237 -bool LLMeshRepoThread::fetchMeshDecomposition(const LLUUID& mesh_id)
   1.238 -{ //protected by mMutex
   1.239 +bool LLMeshRepoThread::fetchMeshDecomposition(const LLUUID& mesh_id, U32& count)
   1.240 +{
   1.241  	if (!mHeaderMutex)
   1.242  	{
   1.243  		return false;
   1.244 @@ -1076,6 +1226,10 @@
   1.245  			}
   1.246  
   1.247  			//reading from VFS failed for whatever reason, fetch from sim
   1.248 +			if (count >= MAX_MESH_REQUESTS_PER_SECOND || mHttpRequestSet.size() >= sRequestHighWater)
   1.249 +			{
   1.250 +				return false;
   1.251 +			}
   1.252  			int cap_version(1);
   1.253  			std::string http_url = constructUrl(mesh_id, &cap_version);
   1.254  			if (!http_url.empty())
   1.255 @@ -1109,8 +1263,8 @@
   1.256  	return ret;
   1.257  }
   1.258  
   1.259 -bool LLMeshRepoThread::fetchMeshPhysicsShape(const LLUUID& mesh_id)
   1.260 -{ //protected by mMutex
   1.261 +bool LLMeshRepoThread::fetchMeshPhysicsShape(const LLUUID& mesh_id, U32& count)
   1.262 +{
   1.263  	if (!mHeaderMutex)
   1.264  	{
   1.265  		return false;
   1.266 @@ -1166,6 +1320,10 @@
   1.267  			}
   1.268  
   1.269  			//reading from VFS failed for whatever reason, fetch from sim
   1.270 +			if (count >= MAX_MESH_REQUESTS_PER_SECOND || mHttpRequestSet.size() >= sRequestHighWater)
   1.271 +			{
   1.272 +				return false;
   1.273 +			}
   1.274  			int cap_version(1);
   1.275  			std::string http_url = constructUrl(mesh_id, &cap_version);
   1.276  			if (!http_url.empty())
   1.277 @@ -1290,7 +1448,7 @@
   1.278  
   1.279  //return false if failed to get mesh lod.
   1.280  bool LLMeshRepoThread::fetchMeshLOD(const LLVolumeParams& mesh_params, S32 lod, U32& count)
   1.281 -{ //protected by mMutex
   1.282 +{
   1.283  	if (!mHeaderMutex)
   1.284  	{
   1.285  		return false;
   1.286 @@ -2249,7 +2407,7 @@
   1.287  			for (U32 i = 0; i < LLModel::LOD_PHYSICS; ++i)
   1.288  			{
   1.289  				// figure out how many bytes we'll need to reserve in the file
   1.290 -				std::string lod_name = header_lod[i];
   1.291 +				const std::string & lod_name = header_lod[i];
   1.292  				lod_bytes = llmax(lod_bytes, header[lod_name]["offset"].asInteger()+header[lod_name]["size"].asInteger());
   1.293  			}
   1.294  		
   1.295 @@ -3047,6 +3205,7 @@
   1.296  			std::set<LLUUID>::iterator iter = mLoadingPhysicsShapes.find(mesh_id);
   1.297  			if (iter == mLoadingPhysicsShapes.end())
   1.298  			{ //no request pending for this skin info
   1.299 +				// *FIXME:  Nothing ever deletes entries, can't be right
   1.300  				mLoadingPhysicsShapes.insert(mesh_id);
   1.301  				mPendingPhysicsShapeRequests.push(mesh_id);
   1.302  			}
     2.1 --- a/indra/newview/llmeshrepository.h	Thu Jun 27 13:55:05 2013 -0400
     2.2 +++ b/indra/newview/llmeshrepository.h	Thu Jun 27 20:38:18 2013 -0400
     2.3 @@ -365,15 +365,15 @@
     2.4  
     2.5  	//send request for skin info, returns true if header info exists 
     2.6  	//  (should hold onto mesh_id and try again later if header info does not exist)
     2.7 -	bool fetchMeshSkinInfo(const LLUUID& mesh_id);
     2.8 +	bool fetchMeshSkinInfo(const LLUUID& mesh_id, U32& count);
     2.9  
    2.10  	//send request for decomposition, returns true if header info exists 
    2.11  	//  (should hold onto mesh_id and try again later if header info does not exist)
    2.12 -	bool fetchMeshDecomposition(const LLUUID& mesh_id);
    2.13 +	bool fetchMeshDecomposition(const LLUUID& mesh_id, U32& count);
    2.14  
    2.15  	//send request for PhysicsShape, returns true if header info exists 
    2.16  	//  (should hold onto mesh_id and try again later if header info does not exist)
    2.17 -	bool fetchMeshPhysicsShape(const LLUUID& mesh_id);
    2.18 +	bool fetchMeshPhysicsShape(const LLUUID& mesh_id, U32& count);
    2.19  
    2.20  	static void incActiveLODRequests();
    2.21  	static void decActiveLODRequests();

mercurial