| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-22 | |||
| 18:07:57 | clarkb | (also I'll have to dig more to see where it set up stack user) | |
| 18:08:02 | mriedem | idk | |
| 18:16:15 | jlk | mriedem: is all of instance resize handled by the migration code paths now? | |
| 18:16:44 | mriedem | jlk: it always was | |
| 18:16:51 | jlk | mriedem: I could have sworn that in older releases (mitaka/newton) that a resize specifically did some ssh'ing around as the user running the nova process | |
| 18:17:26 | mriedem | resize has always equaled the cold migration code paths, | |
| 18:17:32 | mriedem | only difference is the flavor being the same or not | |
| 18:17:36 | jlk | but maybe that's because we had: live_migration_uri = "qemu+ssh://nova@%s/system?keyfile={{ nova.state_path }}/.ssh/id_rsa" | |
| 18:18:53 | mriedem | dansmith: tests are passing, just doing another run with everything to make sure i'm not missing some failing test | |
| 18:18:56 | clarkb | my understanding when we set this up for multinode testing wasw there isn't any reason to not just use root beacuse libvirt ~= root. Which is why we set up root ssh. But its been a while. I bet jogo remembers | |
| 18:19:39 | jlk | I think our local config was not happy about allowing incoming root ssh | |
| 18:19:41 | mriedem | i think if you tunnel via libvirt between the two nodes then that might be ok, but i don't think we're tunneling | |
| 18:19:43 | jlk | so we shunted it to nova | |
| 18:19:55 | mriedem | in devstack i mean | |
| 18:20:18 | mriedem | ganlaksh: hi, i saw your email, sorry i've been busy and haven't had a chance to respond | |
| 18:20:53 | mriedem | ganlaksh: https://docs.openstack.org/nova/latest/#for-contributors would be good to step through if you haven't yet | |
| 18:20:59 | mriedem | note that some of it might be outdated | |
| 18:22:59 | ganlaksh | mriedem: Thanks! Yes I have been going through these docs .. Will continue to look into it.. | |
| 18:27:04 | mriedem | ganlaksh: are you interested in working on a specific part of nova? | |
| 18:27:31 | mriedem | also, what timezone are you in? | |
| 18:28:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Allocate resources on forced dest host during live migration https://review.openstack.org/496031 | |
| 18:28:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove source node allocation after live migration completes https://review.openstack.org/496032 | |
| 18:28:40 | mriedem | dansmith: cfriesen_: here we are ^ | |
| 18:34:01 | ganlaksh | mriedem: I am in the PT zone.. with respect to my areas of interest . i dont have any specific right now.. I would like to pick up some bugs and go through the process.. from the top of my mind.. cells,live-migration, Vmware would be some areas I would be interested in.. I am looking at the tracking page here for the subteams . | |
| 18:35:12 | mriedem | ok, vmware hasn't had much activity in awhile, so if you could dig through some vmware-related bugs and see if they are still valid that would be helpful | |
| 18:35:13 | ganlaksh | mriedem: https://etherpad.openstack.org/p/pike-nova-priorities-tracking. tracking page that i referenced | |
| 18:35:45 | mriedem | ganlaksh: ok that's likely stale at this point | |
| 18:35:58 | mriedem | pike is about to be released, so we'll start focusing on the next release, which is named queens | |
| 18:36:13 | mriedem | ganlaksh: here are open vmware bugs https://bugs.launchpad.net/nova/+bugs?field.tag=vmware | |
| 18:37:06 | mriedem | i don't really know of any easy things for live migration or cells v2 to get started; there were some blueprints related to live migration in pike that had approved design specs but the owners couldn't work on them so they didn't get done | |
| 18:37:44 | mriedem | specifically https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/live-migration-force-after-timeout.html and https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/live-migration-per-instance-timeout.html | |
| 18:38:15 | mriedem | we know we're going to need to work on all of these items for cells v2 in queens https://docs.openstack.org/nova/latest/user/cellsv2_layout.html#operations-requiring-upcalls | |
| 18:38:38 | ganlaksh | mriedem:Ok.. sure. thanks ! Well.let me go through these specs.. arent there any bugs (low hanging fruits!) that i can start working on with respect to live-migration and cells. | |
| 18:38:38 | mriedem | those aren't trivial though | |
| 18:38:56 | mriedem | live migration is really never a low hanging fruit bug type thing :) | |
| 18:39:11 | ganlaksh | mriedem : :) | |
| 18:39:23 | mriedem | looks like there are some open low hanging fruit bugs you could look at though https://bugs.launchpad.net/nova/+bugs?field.tag=low-hanging-fruit | |
| 18:40:51 | mriedem | the next upcoming in-person event is the ptg in denver, colorado the week of 9/11 | |
| 18:40:57 | mriedem | topics for nova at the ptg are being listed here https://etherpad.openstack.org/p/nova-ptg-queens | |
| 18:41:18 | mriedem | if you wanted to get an idea of what we'll be thinking about for the next release | |
| 18:41:39 | ganlaksh | mriedem: Thanks ! i will go through these.. | |
| 18:42:02 | openstackgerrit | Merged openstack/nova master: Skip test_rebuild_server_in_error_state for cells v1 https://review.openstack.org/493076 | |
| 18:42:04 | mriedem | ganlaksh: final thing, are you subscribed to the openstack-dev mailing list? | |
| 18:42:28 | ganlaksh | mriedem: yes..i am .. i am getting the mails... | |
| 18:42:42 | mriedem | ok, be sure to setup filters and route things to folders if you don't want to get overwhelmed | |
| 18:42:53 | mriedem | e.g. i have a nova folder and i send things there that have [nova] in the subject | |
| 18:43:35 | ganlaksh | mriedem: One final question .. I have been using only devstack for playing around.. Is that good enough or what is the recommended thing to do for working on bugs .. | |
| 18:43:52 | ganlaksh | mriedem: Sure will have a folder to get all nova stuff.. | |
| 18:45:34 | mriedem | devstack is good enough | |
| 18:45:48 | mriedem | if you want to poke around on the CLI and test things out in a single-node env | |
| 18:46:27 | ganlaksh | mriedem: Ok cool ! | |
| 19:00:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove source node allocation after live migration completes https://review.openstack.org/496032 | |
| 19:02:44 | mriedem | cfriesen_: were you working on this? https://bugs.launchpad.net/nova/+bug/1712210 | |
| 19:02:45 | openstack | Launchpad bug 1712210 in OpenStack Compute (nova) "Live migration does not restrict to the original cell" [Medium,Triaged] - Assigned to Chris Friesen (cbf123) | |
| 19:02:54 | mriedem | if not i'll fix it quick | |
| 19:05:31 | mriedem | dansmith: something we likely don't want to think about, but if we allocate resources in placement for the dest host during live migration but then something fails and the instance never makes it to the dest node, we aren't cleaning up those allocations anywhere | |
| 19:06:21 | mriedem | the periodic task would have, but that's no longer doing it once everything is upgraded | |
| 19:07:29 | mriedem | i reckon once have the allocations counted against a migration uuid, we could put a periodic in compute that checks for failed migrations where dest_compute == CONF.host and removes any leftover allocations | |
| 19:12:21 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: config drive https://review.openstack.org/409404 | |
| 19:12:31 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: config drive https://review.openstack.org/409404 | |
| 19:14:54 | cdent | mriedem: migration uuid will fix everything! | |
| 19:15:10 | mriedem | i hope so | |
| 19:15:16 | cdent | _everything_ | |
| 19:16:35 | mriedem | https://bugs.launchpad.net/nova/+bug/1712411 for tracking | |
| 19:16:36 | openstack | Launchpad bug 1712411 in OpenStack Compute (nova) "Allocations may not be removed from dest node during failed migrations" [Undecided,New] | |
| 19:17:15 | mriedem | i'm not sure we'd even need the allocation tracked against the migration uuid | |
| 19:17:55 | mriedem | the migration record has a status (error), dest_compute, and instance_uuid in it, that's basically all we'd need for a periodic task in the compute to look for failed migrations targeted themselves and remove the allocations for the dest node and the instance involved in the migration | |
| 19:18:25 | mriedem | using scheduler report client remove_provider_from_instance_allocation | |
| 19:34:51 | cdent | mriedem: your attention to detail is appreciated | |
| 19:38:34 | dansmith | mriedem: the allocation for the migration_uuid can be cleanup-able by whoever doesn't have the instance after the failure, | |
| 19:38:59 | dansmith | and we could have a nova-manage command (or something) that would look at all the active migrations, see which are stalled/failed, and make sure to clean up the allocations for them or something | |
| 19:39:01 | dansmith | but yeah | |
| 19:39:20 | mriedem | sure it doesn't really matter who does it | |
| 19:43:31 | mriedem | things also get weird when you have the ability to cancel an in-progress migration | |
| 19:45:36 | mriedem | if only we had, like, a periodic task in the compute service that would, like, automatically heal the allocations.... | |
| 19:45:40 | mriedem | o-) | |
| 19:51:38 | cburgess | @dansmit Do I recall correctly something about an issue with flavor migration on upgrade to N or O around the created_at, deleted_at, and updated_at columns? I feel like you mentioned that to me at a summit or PTG or something? | |
| 19:51:45 | cburgess | @dansmith even ^^^^ | |
| 19:52:02 | dansmith | um | |
| 19:52:29 | dansmith | cburgess: which flavor migration are you talking about? the more recent one with moving them from the nova to api database? | |
| 19:52:46 | cburgess | Yeah the code in nova/objects/flavor in Ocata. | |
| 19:52:59 | cburgess | dansmith Though it looks like Newton does the same thing. | |
| 19:53:02 | cburgess | Maybe.. | |
| 19:53:20 | dansmith | maybe that there is no soft delete in the api database? | |
| 19:53:29 | mriedem | there would be no deleted_at column either | |
| 19:53:40 | mriedem | because of what dan said | |
| 19:54:25 | mriedem | cburgess: you see it in newton b/c it was added in newton | |
| 19:54:49 | mriedem | https://github.com/openstack/nova/commit/e05acd2005d22556918a60a7621e463b593c34e6 | |
| 19:55:03 | cburgess | Right ok.. | |
| 19:55:54 | cburgess | So we are getting a mysql exception when we try and do the online migration. Its due to the data time formant. The DB has it stored as ISO formant but the migration is expecting it in mysql datetime it looks like. Does that ring a bell? | |
| 19:55:58 | cburgess | Like we missed a step? | |
| 19:56:45 | cburgess | Feels like we have to be missing something because this is too obvious a bug. | |
| 19:57:00 | mriedem | seems like someone was in here talking about something similar a couple of weeks ago | |
| 19:57:35 | cburgess | Maybe @kbringard who is hitting it on our side? | |
| 19:57:49 | kbringard | http://i.imgur.com/SMI1LKj.gif | |
| 19:59:24 | kbringard | @cburgess what now? | |
| 19:59:29 | cburgess | @kbringard well seems like @mriedem and @dansmith don't recall anything like this so... file a bug and propose your patch. Assuming we are right and it merges it would be worth of a backport to the stable branches. | |
| 20:00:08 | cburgess | I need to stop @ | |
| 20:00:09 | cburgess | Stupid twitter habbit. | |
| 20:01:19 | dansmith | cburgess: kbringard yeah I don't know anything about a date format issue | |