| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-22 | |||
| 18:35:12 | mriedem | ok, vmware hasn't had much activity in awhile, so if you could dig through some vmware-related bugs and see if they are still valid that would be helpful | |
| 18:35:13 | ganlaksh | mriedem: https://etherpad.openstack.org/p/pike-nova-priorities-tracking. tracking page that i referenced | |
| 18:35:45 | mriedem | ganlaksh: ok that's likely stale at this point | |
| 18:35:58 | mriedem | pike is about to be released, so we'll start focusing on the next release, which is named queens | |
| 18:36:13 | mriedem | ganlaksh: here are open vmware bugs https://bugs.launchpad.net/nova/+bugs?field.tag=vmware | |
| 18:37:06 | mriedem | i don't really know of any easy things for live migration or cells v2 to get started; there were some blueprints related to live migration in pike that had approved design specs but the owners couldn't work on them so they didn't get done | |
| 18:37:44 | mriedem | specifically https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/live-migration-force-after-timeout.html and https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/live-migration-per-instance-timeout.html | |
| 18:38:15 | mriedem | we know we're going to need to work on all of these items for cells v2 in queens https://docs.openstack.org/nova/latest/user/cellsv2_layout.html#operations-requiring-upcalls | |
| 18:38:38 | ganlaksh | mriedem:Ok.. sure. thanks ! Well.let me go through these specs.. arent there any bugs (low hanging fruits!) that i can start working on with respect to live-migration and cells. | |
| 18:38:38 | mriedem | those aren't trivial though | |
| 18:38:56 | mriedem | live migration is really never a low hanging fruit bug type thing :) | |
| 18:39:11 | ganlaksh | mriedem : :) | |
| 18:39:23 | mriedem | looks like there are some open low hanging fruit bugs you could look at though https://bugs.launchpad.net/nova/+bugs?field.tag=low-hanging-fruit | |
| 18:40:51 | mriedem | the next upcoming in-person event is the ptg in denver, colorado the week of 9/11 | |
| 18:40:57 | mriedem | topics for nova at the ptg are being listed here https://etherpad.openstack.org/p/nova-ptg-queens | |
| 18:41:18 | mriedem | if you wanted to get an idea of what we'll be thinking about for the next release | |
| 18:41:39 | ganlaksh | mriedem: Thanks ! i will go through these.. | |
| 18:42:02 | openstackgerrit | Merged openstack/nova master: Skip test_rebuild_server_in_error_state for cells v1 https://review.openstack.org/493076 | |
| 18:42:04 | mriedem | ganlaksh: final thing, are you subscribed to the openstack-dev mailing list? | |
| 18:42:28 | ganlaksh | mriedem: yes..i am .. i am getting the mails... | |
| 18:42:42 | mriedem | ok, be sure to setup filters and route things to folders if you don't want to get overwhelmed | |
| 18:42:53 | mriedem | e.g. i have a nova folder and i send things there that have [nova] in the subject | |
| 18:43:35 | ganlaksh | mriedem: One final question .. I have been using only devstack for playing around.. Is that good enough or what is the recommended thing to do for working on bugs .. | |
| 18:43:52 | ganlaksh | mriedem: Sure will have a folder to get all nova stuff.. | |
| 18:45:34 | mriedem | devstack is good enough | |
| 18:45:48 | mriedem | if you want to poke around on the CLI and test things out in a single-node env | |
| 18:46:27 | ganlaksh | mriedem: Ok cool ! | |
| 19:00:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove source node allocation after live migration completes https://review.openstack.org/496032 | |
| 19:02:44 | mriedem | cfriesen_: were you working on this? https://bugs.launchpad.net/nova/+bug/1712210 | |
| 19:02:45 | openstack | Launchpad bug 1712210 in OpenStack Compute (nova) "Live migration does not restrict to the original cell" [Medium,Triaged] - Assigned to Chris Friesen (cbf123) | |
| 19:02:54 | mriedem | if not i'll fix it quick | |
| 19:05:31 | mriedem | dansmith: something we likely don't want to think about, but if we allocate resources in placement for the dest host during live migration but then something fails and the instance never makes it to the dest node, we aren't cleaning up those allocations anywhere | |
| 19:06:21 | mriedem | the periodic task would have, but that's no longer doing it once everything is upgraded | |
| 19:07:29 | mriedem | i reckon once have the allocations counted against a migration uuid, we could put a periodic in compute that checks for failed migrations where dest_compute == CONF.host and removes any leftover allocations | |
| 19:12:21 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: config drive https://review.openstack.org/409404 | |
| 19:12:31 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: config drive https://review.openstack.org/409404 | |
| 19:14:54 | cdent | mriedem: migration uuid will fix everything! | |
| 19:15:10 | mriedem | i hope so | |
| 19:15:16 | cdent | _everything_ | |
| 19:16:35 | mriedem | https://bugs.launchpad.net/nova/+bug/1712411 for tracking | |
| 19:16:36 | openstack | Launchpad bug 1712411 in OpenStack Compute (nova) "Allocations may not be removed from dest node during failed migrations" [Undecided,New] | |
| 19:17:15 | mriedem | i'm not sure we'd even need the allocation tracked against the migration uuid | |
| 19:17:55 | mriedem | the migration record has a status (error), dest_compute, and instance_uuid in it, that's basically all we'd need for a periodic task in the compute to look for failed migrations targeted themselves and remove the allocations for the dest node and the instance involved in the migration | |
| 19:18:25 | mriedem | using scheduler report client remove_provider_from_instance_allocation | |
| 19:34:51 | cdent | mriedem: your attention to detail is appreciated | |
| 19:38:34 | dansmith | mriedem: the allocation for the migration_uuid can be cleanup-able by whoever doesn't have the instance after the failure, | |
| 19:38:59 | dansmith | and we could have a nova-manage command (or something) that would look at all the active migrations, see which are stalled/failed, and make sure to clean up the allocations for them or something | |
| 19:39:01 | dansmith | but yeah | |
| 19:39:20 | mriedem | sure it doesn't really matter who does it | |
| 19:43:31 | mriedem | things also get weird when you have the ability to cancel an in-progress migration | |
| 19:45:36 | mriedem | if only we had, like, a periodic task in the compute service that would, like, automatically heal the allocations.... | |
| 19:45:40 | mriedem | o-) | |
| 19:51:38 | cburgess | @dansmit Do I recall correctly something about an issue with flavor migration on upgrade to N or O around the created_at, deleted_at, and updated_at columns? I feel like you mentioned that to me at a summit or PTG or something? | |
| 19:51:45 | cburgess | @dansmith even ^^^^ | |
| 19:52:02 | dansmith | um | |
| 19:52:29 | dansmith | cburgess: which flavor migration are you talking about? the more recent one with moving them from the nova to api database? | |
| 19:52:46 | cburgess | Yeah the code in nova/objects/flavor in Ocata. | |
| 19:52:59 | cburgess | dansmith Though it looks like Newton does the same thing. | |
| 19:53:02 | cburgess | Maybe.. | |
| 19:53:20 | dansmith | maybe that there is no soft delete in the api database? | |
| 19:53:29 | mriedem | there would be no deleted_at column either | |
| 19:53:40 | mriedem | because of what dan said | |
| 19:54:25 | mriedem | cburgess: you see it in newton b/c it was added in newton | |
| 19:54:49 | mriedem | https://github.com/openstack/nova/commit/e05acd2005d22556918a60a7621e463b593c34e6 | |
| 19:55:03 | cburgess | Right ok.. | |
| 19:55:54 | cburgess | So we are getting a mysql exception when we try and do the online migration. Its due to the data time formant. The DB has it stored as ISO formant but the migration is expecting it in mysql datetime it looks like. Does that ring a bell? | |
| 19:55:58 | cburgess | Like we missed a step? | |
| 19:56:45 | cburgess | Feels like we have to be missing something because this is too obvious a bug. | |
| 19:57:00 | mriedem | seems like someone was in here talking about something similar a couple of weeks ago | |
| 19:57:35 | cburgess | Maybe @kbringard who is hitting it on our side? | |
| 19:57:49 | kbringard | http://i.imgur.com/SMI1LKj.gif | |
| 19:59:24 | kbringard | @cburgess what now? | |
| 19:59:29 | cburgess | @kbringard well seems like @mriedem and @dansmith don't recall anything like this so... file a bug and propose your patch. Assuming we are right and it merges it would be worth of a backport to the stable branches. | |
| 20:00:08 | cburgess | I need to stop @ | |
| 20:00:09 | cburgess | Stupid twitter habbit. | |
| 20:01:19 | dansmith | cburgess: kbringard yeah I don't know anything about a date format issue | |
| 20:01:33 | dansmith | kbringard: sorry I missed your ping in -dev earlier .. I was out for a bit | |
| 20:01:52 | kbringard | no worries | |
| 20:02:36 | kbringard | the crux of it looks like, the migrate code gets the data from created_at in nova.instance_types and stores it as a datetime.datetime object with TimeZone (like it's supposed to) | |
| 20:02:40 | kbringard | but it's ISO compliant | |
| 20:02:56 | kbringard | so it looks like so: 2017-08-18 16:45:33+00:00 | |
| 20:02:57 | kbringard | as a string | |
| 20:03:14 | kbringard | or datetime.datetime(2017, 8, 10, 16, 50, 6, tzinfo=<iso8601.Utc> as the object | |
| 20:03:43 | kbringard | then, when it calles flavor_flavor_create the it generates an insert query like so: | |
| 20:03:57 | kbringard | INSERT INTO flavors (created_at, updated_at, id, name, memory_mb, vcpus, root_gb, ephemeral_gb, flavorid, swap, rxtx_factor, vcpu_weight, disabled, is_public) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (datetime.datetime(2017, 8, 10, 16, 50, 6, tzinfo=<iso8601.Utc>), None, 18, 'VCO_cisco_metapod_validation_flavor', 2048, 1, 10, 0, 'db6d7ce0-1d85-49ee-8a73-13f6f64bd46e', 0, 1.0, 0, 0, 1)] | |
| 20:04:11 | kbringard | (1292, "Incorrect datetime value: '2017-08-10 16:50:06+00:00' for column 'created_at' at row 1") | |
| 20:04:11 | kbringard | and fails with | |
| 20:04:26 | kbringard | because ISO compliant datetime != mysql compliant datetime | |
| 20:04:53 | kbringard | simply running it through nova.utils.strtime(), which changes the string to 2017-08-18T16:45:33.000000, fixes it | |
| 20:05:46 | kbringard | so I did a dumb hack like this: http://paste.openstack.org/show/619091/ | |
| 20:05:57 | kbringard | which probably isn't the right way to do it | |
| 20:06:19 | kbringard | but it verifies that if you simply change the object to be mysql datetime compliant the problem goes away and it migrates fine | |
| 20:06:42 | kbringard | so I guess the question I'd have for you, dansmith, since it looks like you did a lot of the base object code, specifically the DateTime objects | |
| 20:06:53 | cburgess | @kbringard Did we look at the earlier schema migrates to see if one of them changes the table schema and we some how misses that one? | |
| 20:06:55 | kbringard | is there some other, earlier, place we should modifying the object | |
| 20:06:56 | mriedem | flavor_values['deleted_at'] = utils.strtime(flavor_values['deleted_at']) is wrong | |
| 20:07:06 | mriedem | there is no deleted_at column in the nova_api.flavors table | |
| 20:07:07 | kbringard | cburgess: yea, I looked at it, they're both datetime | |
| 20:07:22 | cburgess | kbringard k... ok well file a bit and propose the patch I guess... lets see what they say. | |
| 20:07:29 | mriedem | maybe "if 'deleted_at' in flavor_values" is just always False so you never hit that | |