| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-22 | |||
| 16:45:12 | mriedem | well look who it is https://www.youtube.com/watch?v=NzWw_uosDnM&index=6&list=PLOuHvpVx7kYksG0NFaCaQsSkrUlj3Oq4S | |
| 16:50:19 | mriedem | alright functional test is passing, time to fix up the unit tests | |
| 16:52:47 | mriedem | melwitt: are you going to sign up for the red hat pike ptg video this time? | |
| 16:53:01 | mriedem | because you should | |
| 16:53:28 | mriedem | you can be like, quotas this, quotas that, something something claims in the scheduler | |
| 17:05:19 | openstackgerrit | Merged openstack/nova master: Add track_instance_changes note in disable_group_policy_check_upcall https://review.openstack.org/490627 | |
| 17:05:45 | openstackgerrit | Merged openstack/nova master: doc: code review considerations for online data migrations https://review.openstack.org/491885 | |
| 17:06:10 | openstackgerrit | Merged openstack/nova master: Fix a wrong link https://review.openstack.org/494109 | |
| 17:10:36 | openstackgerrit | Drew Fisher proposed openstack/nova master: Add language for compute node configuration https://review.openstack.org/489643 | |
| 17:35:56 | mriedem | wtf, this is super helpful https://gist.github.com/mriedem/706bcfbac773660310c38ad319ce7fe2 | |
| 17:36:11 | mriedem | doesn't exist, create it, sorry it already exists, ok give me it, sorry, it doesn't exist, dummy | |
| 17:43:08 | mriedem | oh SmallFakeDriver, you'll be the death of me | |
| 17:57:42 | ganlaksh | test | |
| 18:03:43 | clarkb | mriedem: dansmith https://docs.openstack.org/nova/latest/admin/ssh-configuration.html says that you need to configure the nova user so that nova can ssh to other compute hosts to move disks around. We don't set this up in devstack-gate/devstack multinode jobs. Is this actually required if using libvirt which ssh's as root? | |
| 18:04:41 | jlk | specifically curious about instance resize | |
| 18:05:14 | mriedem | devstack configures some stuff for live migration using ssh here https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/hypervisor-libvirt#L45 | |
| 18:05:21 | mriedem | that option is deprecated though | |
| 18:05:37 | mriedem | you can tunnel but i believe that's discouraged | |
| 18:06:26 | mriedem | i had brought up the live_migration_uri confusion in http://lists.openstack.org/pipermail/openstack-dev/2017-June/117987.html | |
| 18:06:39 | mriedem | because i was trying to remove that from devstack | |
| 18:07:30 | clarkb | oh is it using the stack user? | |
| 18:07:34 | mriedem | yes | |
| 18:07:46 | clarkb | then why do we set up ssh for the root user? | |
| 18:07:57 | clarkb | (also I'll have to dig more to see where it set up stack user) | |
| 18:08:02 | mriedem | idk | |
| 18:16:15 | jlk | mriedem: is all of instance resize handled by the migration code paths now? | |
| 18:16:44 | mriedem | jlk: it always was | |
| 18:16:51 | jlk | mriedem: I could have sworn that in older releases (mitaka/newton) that a resize specifically did some ssh'ing around as the user running the nova process | |
| 18:17:26 | mriedem | resize has always equaled the cold migration code paths, | |
| 18:17:32 | mriedem | only difference is the flavor being the same or not | |
| 18:17:36 | jlk | but maybe that's because we had: live_migration_uri = "qemu+ssh://nova@%s/system?keyfile={{ nova.state_path }}/.ssh/id_rsa" | |
| 18:18:53 | mriedem | dansmith: tests are passing, just doing another run with everything to make sure i'm not missing some failing test | |
| 18:18:56 | clarkb | my understanding when we set this up for multinode testing wasw there isn't any reason to not just use root beacuse libvirt ~= root. Which is why we set up root ssh. But its been a while. I bet jogo remembers | |
| 18:19:39 | jlk | I think our local config was not happy about allowing incoming root ssh | |
| 18:19:41 | mriedem | i think if you tunnel via libvirt between the two nodes then that might be ok, but i don't think we're tunneling | |
| 18:19:43 | jlk | so we shunted it to nova | |
| 18:19:55 | mriedem | in devstack i mean | |
| 18:20:18 | mriedem | ganlaksh: hi, i saw your email, sorry i've been busy and haven't had a chance to respond | |
| 18:20:53 | mriedem | ganlaksh: https://docs.openstack.org/nova/latest/#for-contributors would be good to step through if you haven't yet | |
| 18:20:59 | mriedem | note that some of it might be outdated | |
| 18:22:59 | ganlaksh | mriedem: Thanks! Yes I have been going through these docs .. Will continue to look into it.. | |
| 18:27:04 | mriedem | ganlaksh: are you interested in working on a specific part of nova? | |
| 18:27:31 | mriedem | also, what timezone are you in? | |
| 18:28:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Allocate resources on forced dest host during live migration https://review.openstack.org/496031 | |
| 18:28:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove source node allocation after live migration completes https://review.openstack.org/496032 | |
| 18:28:40 | mriedem | dansmith: cfriesen_: here we are ^ | |
| 18:34:01 | ganlaksh | mriedem: I am in the PT zone.. with respect to my areas of interest . i dont have any specific right now.. I would like to pick up some bugs and go through the process.. from the top of my mind.. cells,live-migration, Vmware would be some areas I would be interested in.. I am looking at the tracking page here for the subteams . | |
| 18:35:12 | mriedem | ok, vmware hasn't had much activity in awhile, so if you could dig through some vmware-related bugs and see if they are still valid that would be helpful | |
| 18:35:13 | ganlaksh | mriedem: https://etherpad.openstack.org/p/pike-nova-priorities-tracking. tracking page that i referenced | |
| 18:35:45 | mriedem | ganlaksh: ok that's likely stale at this point | |
| 18:35:58 | mriedem | pike is about to be released, so we'll start focusing on the next release, which is named queens | |
| 18:36:13 | mriedem | ganlaksh: here are open vmware bugs https://bugs.launchpad.net/nova/+bugs?field.tag=vmware | |
| 18:37:06 | mriedem | i don't really know of any easy things for live migration or cells v2 to get started; there were some blueprints related to live migration in pike that had approved design specs but the owners couldn't work on them so they didn't get done | |
| 18:37:44 | mriedem | specifically https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/live-migration-force-after-timeout.html and https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/live-migration-per-instance-timeout.html | |
| 18:38:15 | mriedem | we know we're going to need to work on all of these items for cells v2 in queens https://docs.openstack.org/nova/latest/user/cellsv2_layout.html#operations-requiring-upcalls | |
| 18:38:38 | ganlaksh | mriedem:Ok.. sure. thanks ! Well.let me go through these specs.. arent there any bugs (low hanging fruits!) that i can start working on with respect to live-migration and cells. | |
| 18:38:38 | mriedem | those aren't trivial though | |
| 18:38:56 | mriedem | live migration is really never a low hanging fruit bug type thing :) | |
| 18:39:11 | ganlaksh | mriedem : :) | |
| 18:39:23 | mriedem | looks like there are some open low hanging fruit bugs you could look at though https://bugs.launchpad.net/nova/+bugs?field.tag=low-hanging-fruit | |
| 18:40:51 | mriedem | the next upcoming in-person event is the ptg in denver, colorado the week of 9/11 | |
| 18:40:57 | mriedem | topics for nova at the ptg are being listed here https://etherpad.openstack.org/p/nova-ptg-queens | |
| 18:41:18 | mriedem | if you wanted to get an idea of what we'll be thinking about for the next release | |
| 18:41:39 | ganlaksh | mriedem: Thanks ! i will go through these.. | |
| 18:42:02 | openstackgerrit | Merged openstack/nova master: Skip test_rebuild_server_in_error_state for cells v1 https://review.openstack.org/493076 | |
| 18:42:04 | mriedem | ganlaksh: final thing, are you subscribed to the openstack-dev mailing list? | |
| 18:42:28 | ganlaksh | mriedem: yes..i am .. i am getting the mails... | |
| 18:42:42 | mriedem | ok, be sure to setup filters and route things to folders if you don't want to get overwhelmed | |
| 18:42:53 | mriedem | e.g. i have a nova folder and i send things there that have [nova] in the subject | |
| 18:43:35 | ganlaksh | mriedem: One final question .. I have been using only devstack for playing around.. Is that good enough or what is the recommended thing to do for working on bugs .. | |
| 18:43:52 | ganlaksh | mriedem: Sure will have a folder to get all nova stuff.. | |
| 18:45:34 | mriedem | devstack is good enough | |
| 18:45:48 | mriedem | if you want to poke around on the CLI and test things out in a single-node env | |
| 18:46:27 | ganlaksh | mriedem: Ok cool ! | |
| 19:00:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove source node allocation after live migration completes https://review.openstack.org/496032 | |
| 19:02:44 | mriedem | cfriesen_: were you working on this? https://bugs.launchpad.net/nova/+bug/1712210 | |
| 19:02:45 | openstack | Launchpad bug 1712210 in OpenStack Compute (nova) "Live migration does not restrict to the original cell" [Medium,Triaged] - Assigned to Chris Friesen (cbf123) | |
| 19:02:54 | mriedem | if not i'll fix it quick | |
| 19:05:31 | mriedem | dansmith: something we likely don't want to think about, but if we allocate resources in placement for the dest host during live migration but then something fails and the instance never makes it to the dest node, we aren't cleaning up those allocations anywhere | |
| 19:06:21 | mriedem | the periodic task would have, but that's no longer doing it once everything is upgraded | |
| 19:07:29 | mriedem | i reckon once have the allocations counted against a migration uuid, we could put a periodic in compute that checks for failed migrations where dest_compute == CONF.host and removes any leftover allocations | |
| 19:12:21 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: config drive https://review.openstack.org/409404 | |
| 19:12:31 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: config drive https://review.openstack.org/409404 | |
| 19:14:54 | cdent | mriedem: migration uuid will fix everything! | |
| 19:15:10 | mriedem | i hope so | |
| 19:15:16 | cdent | _everything_ | |
| 19:16:35 | mriedem | https://bugs.launchpad.net/nova/+bug/1712411 for tracking | |
| 19:16:36 | openstack | Launchpad bug 1712411 in OpenStack Compute (nova) "Allocations may not be removed from dest node during failed migrations" [Undecided,New] | |
| 19:17:15 | mriedem | i'm not sure we'd even need the allocation tracked against the migration uuid | |
| 19:17:55 | mriedem | the migration record has a status (error), dest_compute, and instance_uuid in it, that's basically all we'd need for a periodic task in the compute to look for failed migrations targeted themselves and remove the allocations for the dest node and the instance involved in the migration | |
| 19:18:25 | mriedem | using scheduler report client remove_provider_from_instance_allocation | |
| 19:34:51 | cdent | mriedem: your attention to detail is appreciated | |
| 19:38:34 | dansmith | mriedem: the allocation for the migration_uuid can be cleanup-able by whoever doesn't have the instance after the failure, | |
| 19:38:59 | dansmith | and we could have a nova-manage command (or something) that would look at all the active migrations, see which are stalled/failed, and make sure to clean up the allocations for them or something | |
| 19:39:01 | dansmith | but yeah | |
| 19:39:20 | mriedem | sure it doesn't really matter who does it | |
| 19:43:31 | mriedem | things also get weird when you have the ability to cancel an in-progress migration | |
| 19:45:36 | mriedem | if only we had, like, a periodic task in the compute service that would, like, automatically heal the allocations.... | |
| 19:45:40 | mriedem | o-) | |
| 19:51:38 | cburgess | @dansmit Do I recall correctly something about an issue with flavor migration on upgrade to N or O around the created_at, deleted_at, and updated_at columns? I feel like you mentioned that to me at a summit or PTG or something? | |