| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-25 | |||
| 13:25:13 | sean-k-mooney | spatel: hi o/ | |
| 13:25:13 | spatel | I did debug and here is the source machine log http://paste.openstack.org/show/733042/ | |
| 13:26:45 | spatel | There are some ERROR | |
| 13:27:09 | spatel | something like this - MessagingTimeout: Timed out waiting for a reply to message ID 80fcdfa3d90d4c128284822024982706 | |
| 13:27:19 | sean-k-mooney | the messaging timeout is strange but may just meean the compute node was not back up after your config change | |
| 13:27:49 | spatel | Let me tell you what happened... | |
| 13:27:53 | spatel | 1. build foo instance | |
| 13:28:06 | spatel | 2. did block migration - and it failed with error | |
| 13:28:28 | spatel | 3. tried again and succeeded but vm failed to boot on destination compute node | |
| 13:29:23 | spatel | if you see my log i have break them in two section first one is for first attempt and second block is retried | |
| 13:29:55 | sean-k-mooney | right | |
| 13:30:20 | sean-k-mooney | the second migration looks ok bu i take it you had the same truncated disk on the dest | |
| 13:30:44 | spatel | yes it had only 1MB disk on destination, it didn't copy full disk | |
| 13:31:24 | spatel | are you seeing anything interesting in logs? | |
| 13:32:01 | sean-k-mooney | no it looks relitivly normal | |
| 13:32:08 | spatel | hmm! | |
| 13:32:43 | spatel | Do you need this line in - live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE | |
| 13:33:10 | spatel | I have meeting i will be back in few mi n | |
| 13:33:11 | spatel | min | |
| 13:34:17 | sean-k-mooney | spatel: i think we removed those flags a few releases go from the config | |
| 13:35:06 | sean-k-mooney | mdbooth: are you aware of any bugs related to block_migration with raw images | |
| 13:35:39 | mdbooth | sean-k-mooney: Can you be more specific? | |
| 13:36:00 | sean-k-mooney | mdbooth: the migration is completing but the disk image is truncated on the destination | |
| 13:36:11 | mdbooth | Oh... | |
| 13:36:26 | sean-k-mooney | mdbooth: then on reboot the vm is borked because the disk is curropt | |
| 13:39:15 | mdbooth | block migration == cold migration, right? | |
| 13:39:35 | sean-k-mooney | mdbooth: no live migration with block migrations of disk | |
| 13:39:46 | mdbooth | ... and the guest is fine until reboot? | |
| 13:39:52 | sean-k-mooney | e.g. no shared sotrage and not boot form volume | |
| 13:40:15 | sean-k-mooney | mdbooth: yes the guest is alive after live migration and then if you reboot it it erros out | |
| 13:40:19 | mdbooth | That's improbable. | |
| 13:40:30 | mdbooth | There's something going on there. | |
| 13:40:36 | mdbooth | something else... | |
| 13:40:38 | sean-k-mooney | mdbooth: fine is proably an over statement the kernel is running from ram at that point | |
| 13:40:56 | sean-k-mooney | if you try to use the disk its proably going to break | |
| 13:41:07 | mdbooth | ENOTENOUGHINFO | |
| 13:48:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject forced move with nested source allocation https://review.openstack.org/605785 | |
| 13:50:13 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Run negative server moving tests with nested RPs https://review.openstack.org/604125 | |
| 13:50:45 | sean-k-mooney | mdbooth: by the way looking at http://paste.openstack.org/show/733042/ is it valid to use imagecahing with raw images? | |
| 13:53:13 | mdbooth | sean-k-mooney: I don't know what I'm looking at there, seems truncated. However yes, imagecache is always used. I rewrote all that stuff to make it a whole lot more obvious a couple of years ago, btw, but it didn't land. | |
| 13:53:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Handle allocations consuming only from the child RPs https://review.openstack.org/608298 | |
| 13:55:00 | sean-k-mooney | mdbooth: ok you looking at part of a systemd log for the n-cpus i think. its an openstack ansible deployment so not really sure how they do logging | |
| 13:55:21 | openstackgerrit | Merged openstack/nova master: Drop legacy cold migrate allocation compat code https://review.openstack.org/611970 | |
| 13:55:41 | sean-k-mooney | mdbooth: anywya what it feels like is only the delta disk was copied insteda of the full disk | |
| 13:56:04 | mdbooth | sean-k-mooney: There is no delta disk for raw. | |
| 13:56:15 | sean-k-mooney | when spatel requires i might suggest moving to qcow2 for not raw | |
| 13:56:25 | mdbooth | sean-k-mooney: Is it possible that they've got qcow2 on the source and flat on the dest? | |
| 13:56:36 | mdbooth | Because if so then I'm pretty sure that would break. | |
| 13:57:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |
| 13:57:04 | sean-k-mooney | am maybe i would be very surpried but that could explain things | |
| 13:57:24 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267 | |
| 13:57:29 | mdbooth | The reason being we explicitly only copy the delta, as we copy the backing file explicitly oob. | |
| 13:57:53 | mdbooth | But if the dest was raw/flat there would be no backing file. | |
| 13:57:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847 | |
| 13:57:59 | sean-k-mooney | mdbooth: ill ask for more info when spatel is back on line. they were going to open a bug for this anyway | |
| 13:58:08 | mdbooth | That would be NOTABUG | |
| 13:58:20 | mdbooth | We don't support that. Not that it wouldn't be nice, but we don't. | |
| 13:58:24 | sean-k-mooney | well its a bug that nova is not chekcing for this and refusing to migrate | |
| 13:58:44 | sean-k-mooney | from an api persepcitive the migration completed sucessfully | |
| 13:58:55 | sean-k-mooney | it also deleted the original disk on the souce | |
| 13:59:19 | mdbooth | Did we ever get sticky host aggregates? | |
| 13:59:42 | mdbooth | i.e. we don't care which host aggregate is selected, but whichever one it is we never go anywhere else. | |
| 13:59:44 | sean-k-mooney | e.g. can migrate out of them by defualt without force | |
| 14:00:06 | sean-k-mooney | mdbooth: no that only happens for AZs today | |
| 14:00:29 | sean-k-mooney | at least as far as i am aware | |
| 14:06:19 | spatel | back | |
| 14:07:50 | spatel | sean-k-mooney: Do you want me to file a bug on this issue | |
| 14:08:02 | spatel | so we have more eyeballs | |
| 14:09:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes https://review.openstack.org/613126 | |
| 14:10:55 | sean-k-mooney | yes please. one thing that mdbooth suggested was to check that the raw images are used on both compute hosts | |
| 14:12:18 | sean-k-mooney | spatel: if you are using qcow on the souce node and raw on the dest we think it could cause the issue you are seeing | |
| 14:12:46 | spatel | I am using raw image | |
| 14:13:13 | mdbooth | spatel: On both source and destination computes? | |
| 14:13:45 | openstackgerrit | Merged openstack/nova master: Consider allocations invovling child providers during allocation cleanup https://review.openstack.org/606050 | |
| 14:14:19 | spatel | I think yes.. i think if you move machine from A to B disk alway stay same right? | |
| 14:14:33 | mdbooth | spatel: look in nova.conf | |
| 14:14:44 | spatel | what option you want to check | |
| 14:14:49 | mdbooth | On both source and destination computes | |
| 14:15:11 | mdbooth | images_type | |
| 14:15:42 | spatel | I don't have that option in nova.conf so must be using default one | |
| 14:16:01 | spatel | is default Qcow ? | |
| 14:16:08 | mdbooth | Yes | |
| 14:16:25 | sean-k-mooney | mdbooth: if you have a raw image in glance it gets converted to qcow by defualt correct | |
| 14:16:39 | mdbooth | Actually, default is modified by use_cow config option | |
| 14:16:57 | sean-k-mooney | which defualt to True | |
| 14:16:57 | mdbooth | spatel: Do you have use_cow set on either source or dest? | |
| 14:18:13 | spatel | use_cow where? on nova.conf ( no) | |
| 14:18:31 | mdbooth | spatel: Yes. | |
| 14:18:37 | mdbooth | use_cow_images on both source and dest | |
| 14:18:43 | spatel | I don't have that option in nova.conf | |
| 14:19:06 | spatel | cat /etc/nova/nova.conf | grep use_cow (nothing found) | |
| 14:19:06 | sean-k-mooney | ok so in that chase your vms should be using qcow images when spawned | |
| 14:19:14 | mdbooth | sean-k-mooney: IIRC the flat backend can not convert qcow2 glances images if so configured, but they're always flat (no backend) | |
| 14:19:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix min config value for shutdown_timeout option https://review.openstack.org/613028 | |
| 14:19:37 | mdbooth | spatel: Are you using containers? | |
| 14:19:51 | mdbooth | spatel: i.e. are you sure you're looking in the correct nova.conf | |
| 14:20:05 | spatel | No containers | |
| 14:20:12 | sean-k-mooney | mdbooth: i think the glance iamges are raw not qcow spatel can you do an image show? | |
| 14:20:42 | sean-k-mooney | spatel: is this not an openstack stack ansible deployment. it used to default to lxc contaier no? | |
| 14:21:14 | spatel | disk_format | raw | |