| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-08 | |||
| 08:16:20 | ivve | seems to set itself to 1 | |
| 08:16:39 | ivve | im no good at debugging this so asking for help :) | |
| 08:18:41 | ivve | note: cap is for entire host, not per vm | |
| 08:18:46 | ivve | instance.. | |
| 08:21:13 | lyarwood | sahid: ^ I think bandwidth=0 is unlimited right? | |
| 08:21:23 | lyarwood | https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainMigrateSetMaxSpeed doesn't help | |
| 08:22:04 | ivve | it should be, but my migrations are 1024KB/sec capped | |
| 08:22:32 | ivve | just evacuated a host and watched my bridge.. 1024KB | |
| 08:22:41 | ivve | so it seems to set itself to 1 | |
| 08:22:48 | ivve | 1MB/sec | |
| 08:24:33 | lyarwood | ivve: live_migration_bandwidth isn't set in nova.conf? | |
| 08:24:43 | ivve | is it meant to be set via nova configuration and/or metadata for hosts/instances? | |
| 08:24:51 | ivve | lyarwood: let me check! | |
| 08:25:43 | ivve | kinda stumbled upon this when i updated a pike | |
| 08:25:47 | lyarwood | ivve: https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/driver.py#L6939-L6942 - you should see it logged here as well btw | |
| 08:26:45 | openstack | bug 1783635 in OpenStack Compute (nova) pike "Live migration failed.: AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'" [High,Fix committed] https://launchpad.net/bugs/1783635 - Assigned to Lee Yarwood (lyarwood) | |
| 08:26:45 | ivve | so im guessing more issues will follow, not only the fix in bug 1783635 | |
| 08:27:12 | ivve | well actually im wrong | |
| 08:27:18 | sahid | lyarwood: yes but that can be overidden by the api call migrateToUri | |
| 08:27:20 | ivve | thats for openstack-ansible to fix | |
| 08:27:23 | ivve | i guess | |
| 08:28:08 | lyarwood | ivve: that's the only other fix on top of 16.1.4 to do with migration AFAIK | |
| 08:28:34 | ivve | lyarwood: you are right, the option is not even in nova.conf | |
| 08:29:23 | ivve | setting it now and restarting, sec for confirmation | |
| 08:30:05 | lyarwood | ivve: it should default to 0 | |
| 08:30:17 | lyarwood | ivve: I was trying to ensure it wasn't set to 1 | |
| 08:31:14 | ivve | it wasn't in the libvirt stanza at all | |
| 08:31:14 | lyarwood | sahid: it defaults to 0 there as well so I have no idea where 1 is coming from for ivve | |
| 08:31:21 | ivve | so im guessing defaults to 1 ? :D | |
| 08:31:41 | sahid | 1 is comming is you are using linux bridge | |
| 08:31:46 | sahid | to slow the migration | |
| 08:32:40 | sahid | to let neutron configure the network on destination node | |
| 08:32:49 | sahid | then neutron is sending an event to nova | |
| 08:33:00 | sahid | and the migration can run at full speed | |
| 08:33:43 | lyarwood | https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/driver.py#L6913-L6922 - yup my bad | |
| 08:34:41 | lyarwood | sahid: does LB send events back once it's done? | |
| 08:34:49 | lyarwood | sahid: is that why we are stuck at 1MB/s | |
| 08:36:29 | ivve | yeah im using linux bridges | |
| 08:38:21 | ivve | it seems it never speeds up | |
| 08:39:00 | ivve | i will confirm | |
| 08:41:58 | sahid | there are some fix in neutron as well | |
| 08:42:00 | ivve | perhaps neutron doesn't send the event. did it used to in pike 16.0.16? | |
| 08:42:23 | ivve | aye feels like im missing some dep. fixes :D | |
| 08:42:48 | ivve | i went from 16.0.16 -> 16.0.17 | |
| 08:43:04 | ivve | there is also some issue between CPU i think | |
| 08:43:40 | ivve | different cpu hosts are whining and don't want to migrate at all ;) | |
| 08:44:12 | lyarwood | ivve: which version of neutron do you have? | |
| 08:44:24 | lyarwood | ivve: as sahid says this needs >=11.0.5 | |
| 08:44:35 | lyarwood | ivve: https://review.openstack.org/#/c/510013/ | |
| 08:47:44 | ivve | neutron-server 11.0.6.dev47 | |
| 08:48:18 | ivve | possibly some configuration then? | |
| 08:49:19 | lyarwood | sahid: ^? | |
| 08:49:29 | lyarwood | ivve: are you seeing the debug line logged by the agent? `Port %s not found, will not notify nova.` | |
| 08:54:04 | mdbooth | owalsh: Did you see the Trilio thing on the ml? | |
| 08:55:14 | ivve | checking | |
| 08:58:23 | sahid | ivve: this part is in the agent, did you well upgrade neutron on the compute nodes? | |
| 08:59:50 | ivve | yes it seems to be the same version | |
| 08:59:59 | ivve | 11.0.6.dev47 | |
| 09:00:12 | ivve | ah wait | |
| 09:00:16 | ivve | you mean agent version | |
| 09:01:05 | ivve | all agents pointing to that version | |
| 09:01:12 | ivve | 11.0.6.dev47 | |
| 09:01:35 | ivve | however since i think backport was missed to pike on the migration guest.py perhaps more things were missed? | |
| 09:01:45 | sahid | hum.. | |
| 09:02:06 | sahid | can you confiugre nova.conf in debug adn then start a live migration, then share the logs | |
| 09:02:07 | ivve | i can't see any port not found error messages related to the migrations... | |
| 09:02:18 | ivve | aye | |
| 09:02:40 | sahid | thanks | |
| 09:02:53 | mdbooth | lyarwood: Did I see some scuttlebutt around nominations for membership of the CIX tiger-tag-team? | |
| 09:04:15 | lyarwood | mdbooth: wrong chat window ;) | |
| 09:04:25 | mdbooth | lyarwood: Ah, so it is. | |
| 09:08:49 | ivve | https://hastebin.com/epafobulet.http | |
| 09:10:20 | ivve | and the other | |
| 09:10:21 | ivve | https://hastebin.com/kevuhihuje.coffeescript | |
| 09:10:35 | ivve | the first was receiving end, second was sending | |
| 09:13:24 | ivve | on these two nodes i have configured the setting live_migration_bandwidth = 0 | |
| 09:13:33 | ivve | even if it is assumed default | |
| 09:15:11 | openstackgerrit | Kevin Zhao proposed openstack/nova master: Skip CPU comparison in Aarch64 https://review.openstack.org/589769 | |
| 09:21:39 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/589513 | |
| 09:21:40 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add workaround to stop use of qemu-img by the RT https://review.openstack.org/589567 | |
| 09:34:54 | openstackgerrit | OpenStack Release Bot proposed openstack/osc-placement stable/rocky: Update .gitreview for stable/rocky https://review.openstack.org/589773 | |
| 09:34:55 | openstackgerrit | OpenStack Release Bot proposed openstack/osc-placement stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/589774 | |
| 09:34:56 | openstackgerrit | OpenStack Release Bot proposed openstack/osc-placement master: Update reno for stable/rocky https://review.openstack.org/589775 | |
| 09:45:45 | sahid | ivve: line 217 we can see Nova reonfiguring libvirt to use 0 | |
| 09:47:29 | sahid | can you try to configure live_migration_bandwidth to a large value instead of 0 | |
| 10:01:47 | mardim | Hello guys one question | |
| 10:02:02 | mardim | I booted an instance succesfully | |
| 10:02:26 | mardim | But when I am trying to resize it I get novalid host was found error | |
| 10:02:31 | mardim | I am using PIke | |
| 10:03:01 | mardim | I am seeing the logs in scheduler and the ComputeFilter returns 0 hosts | |
| 10:03:11 | mardim | I have only one compute node | |
| 10:03:17 | mardim | Any idea ? | |
| 10:03:41 | mardim | artom ^ | |
| 10:04:04 | mardim | melwitt, ^ | |
| 10:05:19 | openstackgerrit | Chris Dent proposed openstack/nova master: Add explicit functional-py36 tox target https://review.openstack.org/589825 | |
| 10:05:27 | lyarwood | mardim: is allow_resize_to_same_host=True ? | |
| 10:05:31 | cdent | gibi ^ | |
| 10:05:35 | mardim | Yes it is | |
| 10:06:32 | mardim | And I definitely have enough resources | |
| 10:06:41 | mardim | because my compute node has 16 cpus | |