Representation of Drosophila larval behaviors by muscle activity patterns

How muscle actions are coordinated to realize animal movement is a fundamental question in behavioral study. To obtain the overall muscular activity patterns accompanying behaviors at high spatiotemporal resolution is technically difficult. In this work, we used light sheet microscopy to simultaneously image and analyze the activity, length and orientation of Drosophila larval muscles across body segments at single muscle resolution in nearly free behaviors. For typical behavioral modes such as peristalsis, head cast and turning, larval muscles showed behavioral mode specific activity patterns. Unexpectedly, reorientation of larval head involves muscle tone in the apparently motionless posterior segments. With a STGCN(spatial temporal graph convolution neural network)-Generator model, sequence of larval behavioral poses outlined by morphological patterns of muscles could be accurately predicted based on the time series of ventral but not dorsal muscle activities, and vice versa. Laser ablation of ventral but not dorsal muscles interrupted peristaltic wave and undermined head cast in both frequency and amplitude. Our results provide a simplified muscle activity representation of soft body motion that can be used for probing the key components of animal motor control.


Introduction 1
Animal behaviors are implemented through the coordinated contraction and extension of its body 2 muscles under the control of motor nervous system. To get a full understanding of neural coding of behaviors, 3 how neural signals are encoded in activity pattern of muscle system and how muscle activity is transformed 4 into final motor output need to be known. Compared with the large body of studies on neural representation 5 of behaviors and muscle activities 1, 2 , the muscle activity representation of behaviors at whole body level is 6 largely lacking. In most animals, to obtain an overall muscle activity representation of body movements at 7 single muscle resolution at high enough spatiotemporal resolution is technically daunting: it needs to monitor 8 the activity of all the related muscles in spatial scope varying from as big as the whole body to as small as 9 individual muscles in freely behaving animals, which is usually out of the capability range of current 10 technologies including electrophysiological recording and optic imaging. Nevertheless, in small and 11 transparent animals such as C. elegans, it is possible to monitor the activity of individual neurons and 12 muscles and the overall behavior under a microscope 3,4 . 13 The model animal Drosophila larva is also small and transparent, making it suitable for simultaneous 14 tracking of behavior and activity of neurons and muscles 5, 6 . It possesses three thoracic segments (T1-T3) 15 and nine abdominal segments (A1-A9), with each semi-segment containing 30 body wall muscles in 16 segments A1-A7 that can be visually identified under microscope 6 . Drosophila larva has a rich reservoir of 17 behavioral movement 7 . In recent years, much work has been done to unravel the neural mechanism 18 underpinning cardinal movements such as forward and backward peristaltic crawling at single neuron 19 resolution 8,9,10,11 , left-right balance 12 , bending 13,14 , up-righting 15 , rolling 16 and so on. The action of body wall 20 muscles has also been closely observed especially during peristalsis 8 . The expression of fluorescent proteins 21 in muscles allowed observation of details of movement and deformation of larval segments at single muscle 22 level 17,18,19 . In a recent report, calcium imaging has also been used to monitor the activity of all muscles at 23 single muscle level in one side of A1 and A2 segment during forward and backward peristalsis under a 24 confocal microscope at high temporal resolution 6 . The underlying neural activity pattern was elucidated 25 based on neuromuscular connectivity and the observed temporal profiles of muscle activities. 26 Here in this work, we set up a light sheet system in combination with calcium imaging to monitor the 27 activity of muscle system in a nearly freely behaving 1 st or early 2 nd instar Drosophila larva. We investigated 28 muscle behaviors at single muscle resolution and described dorsal and ventral muscle activity representation 29 of larval forward/backward peristalsis, head cast as well as turning. We found that muscle activity could be 30 highly correlated with morphological properties such as length and orientation angle, especially for ventral 31 muscles. Furthermore, larval pose sequence could be well predicted based on ventral but not dorsal muscle 32 activity sequences using STGCN-Generator model, and vice versa. The requirement of ventral muscles for 33 larval peristalsis and head cast was confirmed by laser ablation of ventral muscles. The establishment of the 34 concave sides were always active (such as concave and convex muscle 15 in A4 and A5 segment in Figure  23 3a and Figure 3-figure supplement 1, Video 7). The concave muscles in segments immediately posterior to 24 bending point generally contracted slightly. The ratio of muscle length to activity was generally lower in 25 concave than in convex muscles in segments immediately posterior to bending point, suggesting that concave 26 muscles were more apt to be active without contraction compared with convex muscles (Figure 3-figure  27 supplement 4). In consistence with this, muscle tone was obviously more frequent on concave side than on 28 convex side in ventral muscles, although such asymmetry was not seen in dorsal muscles (Figure 3e). Thus 29 these muscle tone type of ventral muscle behavior in post-bending-point segments on concave side were 30 closely related to head cast. We hypothesized that the post-bending-point segment muscles contributed to 31 head cast at least in two ways: first, the slight muscle contraction on concave side not only provided part of 32 the pulling force on anterior segments but also saved necessary space for head cast towards concave side; 33 second, muscle tone probably changed the stiffness of muscles and thus could provide stronger support for 34 the moving anterior segments. It was noted that contribution of other muscles to head cast should not be 35 In short, the reorientation of anterior segments in larval head cast involved the seemingly quiet posterior 2 segments. 3

Turning was signaled by ventral muscles but not dorsal muscles. 4
After the head cast, a larva might change its orientation by aligning the posterior segments with the new 5 direction of head. For simplicity, we focused only on the cases in which orientation of head was largely 6 stable after head cast. Different from in peristalsis and head cast, all orientation angles were calculated with 7 respect to that of T3 segment to demonstrate the process of body alignment towards head since the orientation The most anterior segment with absolute value of relative orientation angle above 20° was defined as turning 10 point, similar to the bending point in head cast. The re-alignment of posterior segments with anterior segment, 11 or turning, was accomplished by forward peristalsis. As shown in Figure 4a and Figure 4-figure supplement 12 1, while muscles in all segments were shortened and lengthened periodically, orientation angles in segments 13 posterior to turning point decreased progressively along with the progress of the peristaltic wave. In our 14 experiments, one or two waves of peristalsis were generally required for the larva to completely realign 15 posterior segments with head. In the earlier second last wave, larva mainly adjusted orientation of segments 16 closely posterior to the turning point (usually at A2 or A3) to align with the anterior part ( Figure 4b). After 17 the wave, the turning point was shifted posteriorly usually to A4. In the last wave, larva mainly adjusted 18 orientation of posterior segments such as A5 to A7 (Figure 4b). It should be noted thatwe used the original 19 angles but not the relative angles of the segments to exclude the effect of head movement on relative segment 20 orientation angle (see supplementary File 1A). Therefore, larva adjusted its body segment orientation mainly 21 in the three segments posterior to the turning point in both waves. Most or even all these changes in segment 22 orientation happened in the peristaltic periods that the segments were contracted, no matter in dorsal view 23 or ventral view ( Figure 4c). 24 We next asked how segment reorientation was correlated with muscle contraction. As shown in Figure  25 4d, for ventral muscles in segments posterior to turning point, more of the reorientation happened during 26 muscle shortening in the more anterior segments, while in more posterior segments more of the reorientation 27 happened during muscle lengthening. As the lengthening of posterior segments happened concurrently with 28 the shortening of anterior segments during the forward propagation of peristaltic wave, the reorientation of 29 these segments happened largely at the same time, that was, when the segment close to turning point was 30 being shortened. It was thus likely that orientation change was caused by the contraction of the segments 31 around the turning point. We noticed that after the peristaltic waves, the length of concave muscles was 32 increased in segments close to the turning point so as to reduce the bilateral muscle length difference that 33 reflected the extent of body bending ( Figure 4e). Accordingly, the high basal activity levels of concave 9 muscles were reduced after peristaltic wave while that of convex muscles remained stable at relatively low 1 level ( Figure 4e). The changes in the last wave in most cases were not significant probably because the 2 bending angles were relatively small ( Figure 4e). In dorsal muscles, the change in muscle length asymmetry 3 and basal activity level was not obvious, although segment reorientation was still clear (Figure 4-figure  4 supplement 2). Thus at least for ventral muscles, peristaltic wave realigned the bilateral muscles not only in 5 length but also in activity level. Such bilateral symmetry promoting realignment was likely the driving force 6 of straightening of larval body during turning. One advantage of realization of turning through peristalsis is 7 probably that larva could change its body orientation and crawl forward at the same time. 8 As turning involved asymmetric peristalsis, we asked if the correlation between muscle activity and 9 muscle length and angle could still be seen. While the negative correlation between activity and length of 10 the same muscles was seen like in normal peristalsis ( . We then widened our scope of correlation from between homologous muscles 13 on the same side to between all concave and convex muscles. We found that orientation angles of muscles 14 in segments posterior to turning segment (mostly A3) on both concave and convex side seemed to be 15 positively correlated with the activity of convex muscles in segments posterior to turning point (except for 16 A7 segment) for ventral muscles (Figure 4f). On the other hand, no obvious correlation pattern was found 17 for dorsal muscles (Figure 4-figure supplement 4). The difference in angle-activity correlation in convex and 18 concave muscles was probably caused by some unknown bilateral temporal difference. 19 Therefore, ventral muscles also out performed dorsal muscles in representing turning behavior. 20

Larval pose sequence could be predicted based on ventral muscle activity patterns. 21
Since larval behavioral movement was the consequence of muscle activity, muscle activity patterns 22 should contain enough information to generate the corresponding larval pose sequences. As shown above, 23 muscle length and angles could be well correlated with muscle activity patterns in some cases for dorsal 24 muscle 9/10 and ventral muscle 15/16. Since only limited number of correlations were investigated, there 25 could be many other correlations that had not been discovered. We postulated that muscle activity pattern 26 might be able to represent larval pose which contains information not only including length and orientation 27 angle of each muscle but also the whole spatial pattern of the muscles. We should be able to predict larval 28 pose sequence using muscle activity data. 29 By focusing on activity of each individual muscle and overall pose represented by the set of muscles, 30 we developed a STGCN generator model consisting of graph convolutional neural network (GCN) and 31 Long-Short term memory (LSTM) to translate the time series of muscle activity into temporal sequences of 32 muscle morphological properties (length, width, angle, and center coordinates of muscle 9 and 10 in segment 33 10 T2-A7 plus one pair of similar muscles in A8 segment on dorsal side, and of muscle 15 and 16 in segment 1 A1-A7 plus two pairs of similar muscles in T2 and T3 segment on ventral side), as shown in Figure 5a. The 2 following two facts had been considered in this model: first, the current pose depends on the previous pose 3 and muscle activities; second, the relationships between muscle activities and pose sequences is general 25,26 4 (see supplementary File 1B for details). Given an arbitrary starting pose and subsequent muscle activity 5 sequence in behavioral modes including peristalsis, head cast and turning, the model was able to generate 6 subsequent pose sequences that resemble true pose sequences of more than 6 seconds long, after training 7 with 2000 epochs on observed ventral and dorsal muscle imaging data (Figure 5b). The disparities calculated 8 by Procrustes analysis between the generated and the ground-truth larval pose was also shown. When pose 9 disparities were less than 0.02, the model generated poses were indistinguishable from the realistic ones. 10 From the pose disparity curve and the sampled prediction-real poses, we could see that the pose sequences 11 generated using ventral muscle data closely approximate the sequences of realistic larval poses ( Figure 5b; 12 Video 8-10). On the other hand, the overall pose sequences generated using dorsal muscle data was different 13 from the ground truth ( Figure 5b, Video 11-13). As shown in Figure 5c, pose disparity was significantly 14 increased in pose sequences based on dorsal muscle activity patterns compared with those based on ventral 15 muscle activity patterns. Therefore, the model trained on ventral muscle data was more promising for 16 modeling relationships between muscle activity pattern and corresponding larval pose. 17 On the other hand, we wondered if larval muscle activity patterns could be predicted based on larval 18 pose. As pose could be considered as the consequence of past muscle activity, the prediction of muscle 19 activity was performed in a rewinding manner, which means that the muscle activity sequence was generated 20 from end to start based on correspondingly reversed pose sequences. It could be seen that in behavioral 21 modes such as forward peristalsis, head cast and turn, the pose-sequence-based prediction of ventral muscle 22 activities were very close to the true muscle activities, whereas such predicted dorsal muscle activities often 23 deviated away from the true signals ( Figure 5d). As depicted in Figure 5e, the prediction of mean average 24 error (MAE) of generated calcium activity sequences was significantly smaller in ventral muscles than in 25 dorsal muscles, i.e., prediction of ventral muscle activities based on pose sequence was more accurate than 26 prediction of dorsal muscle activities. 27 Taken together, these prediction results were consistent with our observation that the correlation between 28 activity and orientation angle was more significant in ventral muscles than in dorsal muscle. Thus, the 29 activity pattern of the ventral muscle 15 and 16 was able to recapitulate the activity of the whole muscle 30 system that produce larval pose and behavior, although ventral muscles alone should not be enough to 31 generate normal larval behavior. 32 6. Laser ablation of ventral but not dorsal muscles interrupted larval peristalsis. 33 As larval ventral muscle activity pattern could well represent behavioral movements, they might play 34 11 functional roles in these behaviors. We then tested this hypothesis by ablating these muscles using two 1 photon laser beam 27 (Figure 6a and 6b). When ventral muscle 15 and 16 in two consecutive segments were 2 ablated bilaterally, spontaneous forward peristalsis started at and backward peristalsis induced by gentle 3 brush touch stopped at the segment of lesion, as if the segments posterior to lesion site were paralyzed. We 4 next bilaterally ablated ventral muscle 15 and 16 in single segment from A1 to A7. Forward and backward 5 peristaltic waves indicated by the rate of change in segment area were found to cut off at, or immediately 6 posterior to, the injured segment, when lesion was in segment A4 or A5 (Figure 6c, Video 14, 15). On the 7 other hand, bilateral laser ablation of dorsal muscle 9 and 10 or skin (used as control) in single segment did 8 not affect the propagation of the peristaltic waves ( Figure 6c, Video 14, 16). We quantified the effect of 9 ventral muscle ablation in single segment on completeness of tail-to-head forward peristaltic waves by 10 counting the number of peristaltic waves in T3 and A7 segment that represented head and tail respectively. 11 While ventral muscle lesion in A6 and A7 segment had no effect and those in A1 and A2 segment had 12 occasional interruptive effect on peristaltic wave, such lesion in A3, A4 and A5 segment could always 13 completely interrupt the peristaltic wave ( Figure 6d). These observations confirmed that ventral muscles 14 were indeed required for larval peristalsis. On the other hand, ablation of dorsal muscle 9 and 10 bilaterally 15 in single segment did not impair larval peristalsis, suggesting that they were not required for normal crawling 16 ( Figure 6d). The effect of muscle lesion on peristaltic wave transmission could also be reflected by the extent 17 of segment contraction in segments neighboring the injured segment. As shown in Figure 6e, contraction in 18 two segments posterior to the lesion site in ventrally injured larvae was not as significant as in the anterior 19 segments, while this was not seen in control larvae and larvae subjected to dorsal muscle lesion. This result 20 further support the function of ventral muscles in larval peristalsis. We also checked impact of muscle 21 ablation on larval head cast. Bilateral ablation of ventral muscle 15 and 16 in segment A3 to A5 significantly 22 decreased the frequency of larval head cast induced by gentle touch in the head while bilateral ablation of 23 dorsal muscle 9 and 10 did not ( Figure 6f). Furthermore, the size of head cast was also undermined by 24 ablation of ventral muscles in A3 to A7 but not by that of dorsal muscles in any segment ( Figure 6g). This 25 was also consistent with the result of model prediction on ability of ventral muscles in representing larval 26 behaviors such as head cast. 27 Taken together, these results showed that ventral muscles but not dorsal muscles were required for larval 28 peristalsis and head cast, which was in support of the ability of ventral muscles to represent larval behavioral 29 movements. 30 31

32
In this work, we used a self-built light-sheet microscope to acquire the real time panoramic muscle 12 activity pattern accompanying Drosophila larval behavioral movements. We found that Drosophila larval 1 soft body movements were mediated by intersegmental coordination of muscle activities. At single muscle 2 level, activity of certain muscle was highly correlated with length or orientation of that muscle or other 3 muscles, especially for ventral muscles. Interestingly, ventral but not dorsal muscle activity pattern was 4 sufficient to generate larval pose sequence using a deep neural network model. Furthermore, laser ablation 5 of ventral but not dorsal muscle at segment from A3 to A5 was able to interrupt propagation of peristaltic 6 wave at the segment of lesion and undermine head cast which apparently did not involve the injured segment. 7 Here we presented a STGCN generator model consisting of graph convolution network and long-short 8 term memory to accomplish the recaption of bidirectional translation between muscle calcium sequence and 9 motor behavior sequence with high accuracy. However, much is still required before we can fully understand 10 such highly complex and synergistic processes. For example, as the data used in this study contain only the 11 calcium activities of a subset of the muscles, other factors such as other muscles that could be involved in 12 pose cannot be excluded. Similarly, although the graph convolution used in this model assumes the spatially 13 closest muscles have the greatest degree of interactions, it is possible that there are muscles controlled by 14 the same source but are spatially far from each other. In spite of these confounders, our model's capability to 15 produce accurate bidirectional translation over a subset of muscle activities and overall sequence within 16 arbitrary time frames suggests that a subset of ventral muscles is sufficient to generate overall motor 17 behaviors. It is also worth noting that without comprehensive muscle states and external environment 18 feedback, our model is still capable of resolving pose sequence from calcium activities within a group of 19 individual muscles and restoring these calcium activities from the reverse overall posture sequence. Thus, 20 although there may be complex intrinsic muscle dynamics and higher-level control signals that our model 21 fails to account for, this subset of ventral muscles may be sufficient to reveal the dominant patterns of larval 22 motor behavior. Furthermore, our model can potentially be extended to longer sequence translation tasks 23 with lower errors in the future, by using a larger dataset and teacher-forcing techniques. 24 The finding that larval behavior poses, at lease those observed in the horizontal plane, could be 25 predicted based only on activity of about 30 ventral muscles is especially intriguing. It would necessarily 26 greatly simplify the deciphering of behavioral movement, since we no longer need to analyze the activity 27 information of all ~600 body muscles to get a full depiction of behavioral poses. It is possible that larval 28 poses could be recapitulated by even fewer muscles. For example, the ventral oblique longitudinal muscle 29 15 and 16 could be mutually redundant in preserving larval pose since they are not only morphologically 30 similar, but also belong to the similar activation group in peristalsis in A1 segment (Zarin et al., 2019). 31 Furthermore, as each muscle is innervated by specific motor neurons, it is possible to use the activity patterns 32 of these muscles to probe the specific activity patterns of innervating motor neurons or premotor neurons 33 that represent corresponding larval behavioral movements. Reducing the number of representing muscles or 34 neurons can help us quickly get the key factors underlying larval motor control. 35 13 Although our results of muscle ablation were in consistence with model prediction, explanation at 1 neural mechanical level is still needed. For disruptive effect of ventral muscle ablation on peristalsis, we 2 assume that spatiotemporal seriality in muscle activation is required for normal larval peristalsis. Since 3 muscle ablation does not injure the central nervous system and the CPG (central pattern generator) that drives 4 peristalsis were intact, the disruption of peristaltic wave by ablation of ventral muscles could be due to loss 5 of feedback from ventral muscles to CPG that is required for further propagation of peristaltic wave. Such 6 feedback signal is probably carried by neurons that sense muscle contraction. Similarly, for effect of muscle 7 ablation on larval head cast, we assume that muscle tone of ventral muscles in posterior segments generate 8 neuronal signals that are required for the activation of the neurons commanding larval head cast, so that 9 ablation of these ventral muscles undermines frequency or size of head cast. These hypotheses need to be 10 further confirmed. 11 It is unexpected that activities in ventral muscles are more highly correlated with larval behavioral poses 12 than that in dorsal muscles, since dorsal muscle 9 and 10 are apparently the strongest muscles. This is 13 probably because of the following reasons. First, ventral muscles 15 and 16 are obliquely aligned so that 14 their orientation angles are sensitive to peristaltic contractions that are mainly in longitudinal direction. On 15 the contrary, dorsal muscle 9 and 10 are largely parallel to the longitudinal midline so that their orientation 16 angles are insensitive to peristaltic contraction. Second, ventral muscles except for those in A7 segments 17 span two segments which makes them sensitive to deformation in two adjacent segments that include more 18 pose information. Dorsal muscle 9 and 10 however, span only one segment, so that they are relatively less 19 sensitive to deformation in neighboring segment. Third, ventral muscles in normal cases directly touch 20 supporting ground so that they provide the direct supporting force for larval motion. On the other hand, 21 dorsal muscles do not directly exert force on surrounding environment and they provide auxiliary force for 22 larval movement. Although in microfluid chip larva might touch surrounding parts on both dorsal and ventral 23 sides, exerting force with ventral muscles may still be the more natural way. 24 One of our discovery is the comprehensive occurrence of muscle tone that is equivalent to isometric 25 muscle contraction in mammalian skeletal muscles. In larval peristalsis, muscle tone was frequently seen 26 especially in posterior segments during backward crawling, leading to the rightward skewness of activity 27 curves. In larval head cast, muscle tone in concave ventral muscles always occurs during the reorientation 28 of anterior segments. We propose that in peristalsis muscle tone helps to keep muscles primed and ready for 29 activation, while in head cast muscle tone helps to maintain body balance and postures by changing the 30 stiffness of muscles. 31 One prominent feature of Drosophila larval movement is the soft-style, i.e. continuity and smoothness. 32 Based on our observation, the continuity and smoothness reside not only in the flexibility of the physical 33 structure, but also the spatiotemporal coordination of muscle behaviors. In peristalsis, the temporal range of 34 14 muscle behaviors in neighboring segments overlaps. Meanwhile, temporal sequence is well established in 1 homologous muscles. Considering that lateral muscles behave later than the longitudinal muscle 6 , the 2 temporal overlap should be even larger. Such spatiotemporal continuity in muscle behaviors supports the 3 apparently continuous but not continual physical movement. The spatiotemporal coordination is also 4 reflected by muscle tone. In peristalsis, muscle tone prepared the muscle for upcoming activation when the 5 wave is still in preceding segments, thus facilitating smoother spatiotemporal transition between segments. 6 In head cast, muscle tone with slight muscle shortening in segments posterior to bending point also allow 7 smoother and coordinated reorientation in anterior segments. 8 In summary, our work disclosed the activity patterns of reduced number of muscles for describing larval 9 movements and provided simplified physiological signatures for soft body behaviors at single muscle 10 resolution that could facilitate future investigation of animal motor control. passed through a collimator to form a collimated Gaussian beam with ~3.3mm diameter light spot, and then 19 a lens followed by a cylindrical lens was used to compress the Gaussian light into 1.65mm in height while 20 keeping the width of light unchanged. The light beam was then passed through a second cylindrical lens to 21 further expand the width of the beam from 3.3mm to 9.9 (1.65mm in height). The reshaped elliptical beam 22 was split into two beam paths using a 50/50 beam splitter, with the first beam sequentially passing an optical 23 slit, a cylindrical lens and an illumination objective to form a thin laser sheet that projected onto the sample; 24 and the second beam passing the similar optical modules except for two additional relay lens that could 25 adjust the 2 nd laser sheet to completely align it with the 1 st one from the opposite direction. 26 The fluorescence detection was based on an Olympus MXV10 microscope. To achieve a large enough 27 field of view (FOV), we used a low-magnification 4x objective combined with a Photometrics Iris 15 camera 28 to capture all the tracks of the free movement of a 1 st instar or early 2 nd larva in a 5mm x 0.6mm square 29 chamber of a customized microfluidic chip. The position and orientation of the sample could be adjusted 30 using a customized sample holder. The light-sheet microscope allowed selective-illuminated, high-contrast 31 fluorescence imaging of the moving muscles at a frame rate of 20 frames per second. Since the larva body 32 was thick (~120um, the height of the microfluid chip chamber) and induced the signal scattering from the 1 ventral muscles. We then flipped the chip bottom-up to obtain clear images of the ventral muscles. All the 2 recorded images were processed using ImageJ. 3 Tracking of muscle signal: The dorsal and ventral muscles were manually tracked by defining the four 4 corner points of each muscle rectangle. The calcium intensity in the rectangle was quantified using a Matlab-5 based custom script. Note that although we have tried to control the level of light sheet to scan only muscle 6 9 and 10 on dorsal side, so that the possible partial overlap between areas of muscle 9 and 10 and that of 7 muscle 1 and 2 was minimized. For the rectangle, the middle points between neighboring points were 8 calculated as the average x and y axis values. The longer distance between the opposing middle points were 9 calculated as muscle length while the shorter distance was muscle width. The orientation of the line of muscle 10 length was used as orientation angle of that muscle. The mean orientation of the two bilaterally symmetrical 11 muscles was used as the orientation of each body segment. The orientation angle of segment A7 was 12 subtracted from that of each muscle to obtain the relative orientation angle of each muscle. 13 For judgment of start of each calcium wave or muscle contraction, the minimal calcium level and 14 maximal muscle length in the whole video that encompass at least two periods of peristalsis was defined as 15 basal calcium level and length of an intact muscle. The time point that calcium signal rise to 10% of basal 16 level was defined as the start time of that calcium wave. The time point that muscle length was reduced to 17 be less than 90% of full length was defined as the start time of that contraction. 18 Muscle Laser Ablation: Put a washed clean early 2nd instar larva on a 3% agar plate and cover it with a 19 cover glass. Move the cover glass to roll the larva so that its back or abdomen face the objective of two 20 photon microscope for laser ablation. Under a two-photon microscope (FVMPE-RS, Olympus Inc.) 21 equipped with a 25x objective, find target muscle and place it in the center of field of vision and adjust focus 22 to make the end of muscle to be the clearest. In the software control panel, draw a line at the end of the 23 muscle to be ablated. Line scan the muscle end using an 800nm laser beam at intensity of ~30% of maximal 24 power for ~1 second. Compare the image before and after ablation to ensure that the muscle is ablated. After 25 laser ablation, remove the coverslip and transfer the injured larva onto a new 1.5% agar plate. Record larval 26 behaviors on the agar plate. Backward peristalsis and head cast were induced by gentle touch in larval head 27 with a brush. For head cast, each larva was repeatedly touched for five times. The size of head cast was 28 calculated as the maximal angle between midline of segment T3 and the midline of segment A7 midline. 29 Attempted head casts with maximal angles less than 20 degrees were not counted in. The number of head 30 cast after five times of touch was counted for each larva. 31 Prediction model: A pair of encoder and decoder constructs an encoder-decoder generator neural network 32 capable of mapping larval pose sequences to muscle activity sequences in a one-to-one manner. The encoder 33 stacks two layers of spatial-temporal graph convolution operators and the LSTM layers. The decoder stacks 34 one LSTM layer, two linear layers and two layers of spatial-temporal graph convolution operators. 1 The generator neural network is optimized based on the sequence reconstruction loss and adversarial 2 loss. Reconstruction loss is an L1 loss between generated sequence and real sequence. The adversarial loss 3 is introduced by an extra discriminator. The discriminator shares similar architecture of encoder is used to 4 extract the distinguishable features for sequence. This means that the adversarial loss can be summed up 5 as a difference in distribution between generated and real sequences, estimated using a neural network. 6 Finally, we train the discriminator to maximize the adversarial loss, as well as the generator neural network 7 to minimize the reconstruction loss and adversarial loss. 8 We trained such models from muscle activity to pose and from pose to muscle activity for the dorsal 9 and ventral datasets, respectively. The pose sequence is represented by a tensor ∈ × × corresponding 10 to frames, muscles, and channels of morphologies, the muscle activity sequence is represented by 11 a tensor ∈ × corresponding to frames, muscles, and 1 channel of calcium activity. When 12 predicting the larval pose sequence, an initial pose 0 ∈ × and a corresponding muscle activity 13 sequence 1: ∈ × × are fed into model to generate the predicted pose sequence � 1: ; when predicting 14 the larval muscle activity, a final muscle activity state +1 ∈ and a corresponding reversed pose 15 sequence T:1 ∈ × × are fed into model to generate the predicted reversed muscle activity sequence 16 � T:1 . 17

Software:
The code was implemented with Python, the complete project code is available on GitHub: 18 https://github.com/CSDLLab/motor-behavior-recaption. The database that stores the muscle morphological 19 properties (length, width, angle, center coordinates) and muscle activity have been organized into a uniform 20 format to allow for novel analysis.  to that during forward peristalsis, rate of muscle tone in ventral muscles during backward peristalsis is 7 generally higher than in forward peristalsis. n.s. not significant. *P<0.05, **P<0.01, Sidak's multiple 8 comparison test after two-way ANOVA. Error bars, SD. 9 e. Inter-segmental and intra-segmental muscle activity delay in dorsal and ventral muscles in forward and 10 backward peristalsis. The delays within one segment and between neighboring segments are generally 11 not significantly different. Inter-segmental delays and intra-segmental delays are shown in green and 12 magenta respectively. The labels for intra-segment that is supposed to lie between two neighboring 13 segments are not shown due to limited space. n.s. not significant, * P<0.05, Sidak's multiple comparison 14 test after two-way ANOVA. n = 24 for all cases. Thick line and dot-line in violin plot, median and 15 interquartile range. shown in green and red respectively. The labels for intra-segment that is supposed to lie between two 5 neighboring segments are not shown due to limited space. n.s. not significant. * P<0.05, ** P<0.01, Sidak's 6 multiple comparison test after two-way ANOVA. n = 24 for all cases. Thick line and dot-line in violin plot, 7 median and interquartile range. correlation between length and the activity of the same or neighboring muscle is seen in forward and 4 backward crawling for both dorsal and ventral muscles. Patterns of stronger correlation between activity and 5 orientation angle can be seen in ventral but not in dorsal muscles for both forward and backward peristalsis. 6 Note that the correlation coefficients along the diagonal line in ventral muscles are negative in left muscles 7 and positive in right muscles respectively due to the opposite direction change of ventral muscle orientation 8 angles in left and right during peristalsis. For dorsal muscle 9 and 10, n = 8 waves for forward and n = 6 for 9 backward; for ventral muscle 15 and 16, n = 6 for forward and n = 4 for backward.  c. Representative curves of activity difference between concave and convex muscles and segment midline 10 orientation calculated based on dorsal muscle 9 and ventral muscle 15 during larval head cast. The grey 11 dotted boxes represent periods of head contraction. The green dotted boxed represent periods of re-12 orientation in anterior segments. Note that during head cast stage, the curves of activity difference in A3, 13 A4 and A5 segment, and curves of segment orientation in segments anterior to A4, are monotonically 14 31 increasing. Bending points are generally around A2. 1 d. Mean correlation between concave-convex muscle activity difference and segment orientation angle 2 calculated based on muscle 9 (left) and muscle 15 (right). Segment number 0 is the bending segment. 3 Negative and positive numbers indicate segments anterior and posterior to the bending segment 4 respectively. Strong positive correlation is seen between orientation angle of segments anterior to 5 bending point and bilateral activity difference in segments posterior to bending point. n=6 for dorsal 6 muscle 9, n = 5 for ventral muscle 15. 7 e. Rate of muscle tone in dorsal and ventral muscles on concave and convex side during head cast. Note 8 that in segments posterior to bending segment, rate of muscle tone is generally higher in ventral muscles 9 on concave side than those on convex side. n.s. not significant, *P<0.05, Sidak's multiple comparison 10 test after two-way ANOVA. The dark and light grey areas represent periods of the last waves and the second last wave respectively. 12 Anterior segment re-orientation in A4 and A5 happens mainly in the earlier second last wave, while the 13 last wave mainly changes the orientation of posterior A6 and A7 segment. Note that the change of 14 orientation angle in different segments largely happens in the same period. In more anterior segments, 15 the reorientation happens during muscle shortening and lengthening. In posterior segments, the 16 reorientation happens mainly during muscle lengthening. turning point is at around A3 before the second last wave and around A5 before the last wave. * P<0.05, 20 ** P < 0. 01, *** P < 0.001, paired t-test. 21 f. Mean correlation between muscle activity and length and orientation angle of ventral muscle 15 on 22 concave and convex side. Strong negative correlation is seen between activity and length of the same 23 muscles (top row). Correlation pattern is seen between convex muscle activity and orientation angle of 24 muscles on both sides (middle and bottom row). n = 5. Change of bilateral length difference in dorsal muscle 9, activity of muscle 9 on 1 concave and convex side, and segment midline orientation after the last and second last peristaltic wave 2 during turning. The bending segment is at around A3 before the second last wave and around A5 before the 3 last wave. * P<0.05, ** P < 0. 01, paired t-test.    c. Bilateral laser ablation of the ventral muscles 15 and 16, but not dorsal muscle 9 and 10, interrupts both 9 forward and backward peristaltic waves. The long yellow arrows indicate the peristaltic waves in 10 segment area. The magenta dotted rectangles mark the segment of muscle ablation. In control, only skin 11 but not muscles are injured. Note the absence of peristaltic wave in segments posterior to injury in 12 ventral muscle ablation. 13 47 d. Comparison of peristaltic wave numbers seen in anterior T3 and posterior A7 segments in larvae 1 subjected to muscle ablation at different segments within two minutes after injury. Bilateral ablation of 2 ventral muscles in A3 to A5 segment completely abolished the peristaltic wave in A7 segment. Ablation 3 of dorsal muscle in all other segments has no obvious effect on completeness of wave propagation. 4 e. The maximal contraction during peristaltic wave, measured by the ratio of minimal segment area to full 5 segment area, is reduced in segments posterior to ventral muscle ablation. Segment contraction anterior 6 and posterior to the injury is not affected by ablation of dorsal muscles. Injured segment is the segment 7 of muscle ablation. Positive and negative numbers indicate segments posterior and anterior to the injury. 8 f. Ablation of ventral but not dorsal muscles reduces the frequency of larval head cast. Number of head 9 cast is counted after five times repeated touch in the head for each larva.   2e, S2c Muscle contraction the length of muscle is less than 0.9 times the maximal 2c 4 length for each muscle Muscle tone the length of muscle is longer than 0.9 times the basal length and the calcium signal is higher than 1.5 times basal calcium signal, that is, the minimal calcium signal for each muscle.
Muscle relaxed the length of muscle is more than 0.9 times the basal length and the calcium signal is less than 1.5 times basal calcium signal, that is, the minimal calcium signal for each muscle. The turning point the anterior most segment with absolute value of orientation angle (relative to T3) above 20° during turning.

4c
The last wave the peristaltic wave that makes larval tail completely aligned with head.

S4b
The second last wave the peristaltic wave before the last wave. The size of head cast the maximal angle between T3 midline and A7 midline, which must greater than 20 degrees. Similarly, the entire muscle activity sequence feature matrix ∈ × ×1 stacks × calcium intensity values.
To accomplish the rational translation between muscle activity and pose sequence , the target sequence generation follows the differential equation: where Δ is a hyper-parameter indicating the length of historical information used in predicting changes at next step. represents a generator neural network that captures the internal interplay patterns between muscles and estimates the effects of muscle activity on pose changes. This generator neural network is implemented with encoder-decoder architecture, and trained in an adversarial manner.
Network architecture. There are a generator network and a discriminator network used in training stage. The generator network learns translation between muscle activities and pose sequence, and the discriminator network learns to distinguish whether target sequence is generated by the generator network or sampled from real data. Incorporating the adversarial generator-discriminator training enhance the performance of generator network. The generator neural network consists of encoder and decoder. The encoder network is composed of two layers of spatial-temporal graph convolution operators and one LSTM layer. The decoder is composed of one LSTM layer, two linear layers and 9 two layers of spatial-temporal graph convolution operators. The spatial-temporal graph convolution operator served as low-level feature extraction/reconstruction and the LSTM layer served as longrange nonlinear time series feature modeling. The encoder takes a frame of muscle pose and a set of subsequent muscle activity sequences as input, encodes the joint patterns of muscle activity and initial pose into context features as input of decoder. Conditioned on the context features, the decoder extracts and reconstructs the subsequent pose changes sequence in autoregressive manner.
The discriminator network employs the same architecture as the encoder, and is trained to maximize the difference between generated and realistic sequences, whereas the generator network is trained to minimize the difference.
Spatial-Temporal graph convolution operator. The spatial-temporal graph convolution stacks two steps: (i) the spatial graph convolution and (ii) the temporal graph convolution. Following Yan al et. 1 the spatial-temporal graph convolution can be achieved using the node adjacent matrix and entire nodes feature matrix as: where ∈ ℝ × is the output of spatial neighbors feature aggregation that contains channels features, ∈ ℝ × denote the identity matrix with dimensions indicating the self-loop of the nodes, ∈ ℝ × is the diagonal degree matrix with = ∑ , the hyper-parameter controls the temporal range to be included in the neighbor graph and can also be referred to as the temporal convolution kernel size, ∈ ℝ × × and ∈ denote the learnable weights and bias respectively. From the view of one node (e.g., node at moment ), the spatial-temporal graph convolution can be regarded as transforming the features of X t,i ∈ ℝ 1× and its spatial-temporal adjacent nodes { , | , ≠ 0, − ≤ ≤ } to ∈ 1× with the shared and , in which the shared learnable parameters are useful to learn the most prominent patterns among all nodes in muscles interactions. In practice, a complete spatial-temporal graph convolution operator is implemented with standard 2D convolution and multiplies results with normalized adjacent matrix on the node number dimension.
Encoder-Decoder generator neural network. The generator neural network seeks to model the conditional probability of the output sequence given the input initial pose and muscle activity sequence, i.e., ( 0 , 0 , … , −1 ).
We applied spatial-temporal graph convolution on each node to capture the correlation of muscles across spatial and temporal dimensions. The features from spatially adjacent nodes and the features from the same node in consecutive poses will be fused and propagated forward repeatedly in a graph convolutional neural network iteratively (illustrated in Figure 6a). After feedforwarding of several layers, the model was capable of capturing the relationship patterns of entire graph sequence.
Encoder. The encoder stacks two layers of spatial-temporal graph convolution operators to double the input sequence feature channel for capturing the correlation of muscles across spatial and temporal dimensions. The features from spatially adjacent nodes and the features from the same node in consecutive poses will be fused and propagated forward spatial-temporal graph layers as the input of LSTM layers. Then the LSTM layers were used to model the dynamics of sequential features. Where the original input is combination of a muscle activity sequence feature matrix and an initial pose. We first duplicated the initial pose times to form a sequence, then concatenated two sequences along the feature channel dimension as input. By processing the entire input sequences, the final hidden states of LSTM encoded the overall muscle activity patterns. These hidden states were the initial states of decoder LSTM. The output feature sequence was the input to decoder.
Decoder. The decoder stacks one LSTM layer, two linear layers and two layers of spatial-temporal graph convolution operators aims to extract pose dynamics features from the encoder output sequence and reconstruct it into pose sequence. The decoding of decoder works as: where is the output of decoder LSTM layer at moment, � is the generated pose at time step, MLPs(⋅) is stacked two linear layers, [] represents the feature concatenation operation, and − (⋅) is stacked graph convolutional operators for mapping latent features into muscle states displacements (i.e., velocity features) based on historical poses and muscle activity information.
Put it all together, we employed an encoder-decoder architecture to translate a long-term muscle activity sequence into larval pose sequence.

11
Discriminator. To train a model that captures the rich internal dynamics of joint muscle states and external control signals, we used a generative adversarial training approach. The generator neural network is referred to as a and a discriminator for the purpose of simplicity. The discriminator has similar architecture used by the encoder of generator. The ST-GCN is used for high-level features extraction followed by LSTM layers for modeling long term patterns of muscle states and external control signals jointly. Using the discriminator model, the distance between the generated fake data and the real data is as follows: where indicates the distribution of real muscle states, indicates the distribution of generated muscles states, ‖ ‖ ≤ 1 means that which indicates the discriminator function has to be 1-Lipschitz (or -Lipschitz for some constant ).
Adversarial Training. Given the generator and discriminator, our LSTM Encoder-Decoder Dynamic System aims to minimize the difference between generated muscle states and realistic muscle states: where the adversarial loss equivalent to the Wasserstein distance between the realistic data distribution and the generated fake data distribution, [•] indicates the expectation, the generator aims to cheat the discriminator thus minimizing the loss, while the discriminator aims to discriminate the data is real or fake thus maximizing the loss. Besides that, in order to improve the realistic of visual effects of generated muscle states, we added a reconstruction loss for the sequential states reconstruction, applying the L1 loss between all the generated muscle states and realistic muscle states: = �∥ � 1: − 1: ∥ 1 �, where the [•] indicates the expectation, the reconstruction loss indicates the overall morphological difference between the generated pose sequence � 1: and the realistic pose sequence 1: .
To this end, the complete training loss is: where weights the reconstruction term.

Translation between muscle activities and pose sequences
Data preprocess. The muscle states data were collected from individuals Drosophila larvae with calcium imaging, including the muscles on the dorsal side and ventral side. On the dorsal side, 38 clear and bilaterally symmetric muscles were selected. On the ventral side, 30 clear and bilaterally symmetric muscles were selected. The rectangular masks are used in muscle area labeling based on the geometric characteristic of muscles. The dorsal side dataset included 11 individuals and counts over 1200 frames in total. The ventral side dataset included 15 individuals and counts over 1400 frames in total. The datasets were formed by original TIFF image sequences and the corresponding json labels. The muscle area representation was transformed from the original cartesian coordinates of rectangle vertices into length, width, angle, and center coordinates. The average calcium intensity in muscle area was measured as the activity level of the muscle. All calcium activity values were pre-normalized to the range of [0, 1].
Implementation. All our models were implemented with Pytorch and trained by WGAN-Clip, using RMSprop optimizer with learning rate 0.0001, the clipping value is 0.01 and the value is 0.01.
To handle the different shapes and calcium levels of individuals, all muscle states and calcium intensities were normalized according to the mean and standard values of the whole dataset.
Training. If there are no special instructions, all training is done in generator-discriminator adversarial training manner for 2000 epochs. And the batch size is 4, the lengthes of all training sequences are fixed with = 64 and the sampling gap between consecutive poses is 100ms.
Muscle activities to pose sequence translation. Long-term pose sequence prediction aimed to translate muscle activities into corresponding pose sequence over 6400 milliseconds, which was challenging due to accumulated estimation errors and nonlinearity biochemical state variations. Let X be the morphological features (length, width, angle, and the center coordinates) of pieces of the muscles, and let U be the muscle activities, and the initial 0 as arbitrary pose at any moment, then train the model on ventral data and dorsal data separately. For ventral data, the ∈ ℝ × × and ∈ ℝ × × , conditioned on the initial pose ∈ ℝ × , the spatial-temporal graph convolution layers of encoder learn to capture the patterns of relationship between individual muscle activities, and the LSTM layer learns to model the long-term muscle activities patterns. Then, in the decoder, the muscle activity patterns from encoder is used to navigate the autoregressive generation of 13 individual muscle hidden features. Finally, with linear layers and spatial-temporal graph convolution layer, hidden features are reconstructed back into morphological features. For dorsal data, the only differences were ∈ ℝ × × and ∈ ℝ × × .
Pose sequence to muscle activities translation. In task translating muscle activity sequence into pose sequence, we followed the concept that future pose is determined by past muscle activities. Thus, in task translating pose sequence into muscle activity sequence, we also follow the concept that past muscle activities is determined by future pose. Extrapolating the muscle calcium activity sequence from the pose sequence can be regarded as the rewind of the task of extrapolating the larval pose change sequence from the muscle activity sequence. This is also challenging due to accumulated estimation errors and nonlinearity biochemical state variations. For ventral data, suppose ∈ ℝ × × are the muscle activities matrix, ∈ ℝ × × are morphological features (length, width, angle, and center coordinates) matrix, and the initial 0 is muscle activity states at the moment after the last moment of chose muscle activity sequence, then flip the and along the temporal dimension. In the view of training, the only difference between the translation of muscle activity sequence to pose sequence and the translation of pose sequence to muscle activity sequence is the definition of input and output.

Pose disparity metrics:
To evaluate the appearance difference between estimated muscle behaviors and ground truth muscle behaviors, we transformed muscle representation from node features into cartesian coordinates. Each piece of muscle was converted from morphological features (length, width, angle, center coordinates) to coordinates of four rectangle vertices. We used the Procrustes analysis between the estimated muscle vertices key-points and corresponding ground-truth keypoints as appearance comparison metric. In Procrustes analysis, each input matrix was considered as a set of points (the rows of the matrix). The difference between the shape of two poses were evaluated after "superimosing" the two shapes by translating, scaling and optimally rotating them.
Then, the square root of the transformed shapes between corresponding points represented the shape disparity,