Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-01
16:13:43 jangutter sean-k-mooney: you guys are OK, but I continually fail my Turing test.
16:14:57 stephenfin I always knew there was something fishy about jungleboyj
16:15:27 stephenfin (RE: "jungleboyj has quit (Killed (Sigyn (Spam is off topic on freenode.)))")
16:15:56 tssurya dansmith: are we having a cells meeting today?
16:16:19 dansmith tssurya: I don't have anything to discuss and am also focused on other things, you?
16:16:24 dansmith mriedem_afk: melwitt ^
16:16:25 tssurya I don't have anything either
16:16:56 melwitt +1 to skip
16:16:59 tbachman /WHOIS Turing^H^H^H^H
16:29:34 dansmith gawd, the testing of resource tracker... ugh
16:32:38 melwitt the ol RT
16:38:11 mriedem_afk dansmith: i had some things posted on the agenda
16:38:31 dansmith mriedem: who looks at that?
16:38:36 mriedem https://wiki.openstack.org/wiki/Meetings/NovaCellsv2#Agenda
16:38:49 mriedem i'll just dump them here
16:38:51 mriedem couple of new bugs
16:38:53 mriedem https://bugs.launchpad.net/nova/+bug/1784074
16:38:53 openstack Launchpad bug 1784074 in OpenStack Compute (nova) "Instances end up with no cell assigned in instance_mappings" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
16:39:10 melwitt gnuoy: yes, it does need access to the api database in order to 'get_metadata_by_instance_id' it looks at the api database to determine what cell the instance is in
16:39:17 mriedem i started https://review.openstack.org/#/c/586713/ for that but need to do more probably, like collapse those 2 for loops as we said a year ago
16:39:37 dansmith mriedem: yeah, I'm not really sure where we are on that.. there are several patches up
16:39:41 mriedem also, mnaser has a script in that bug for healing some of these types of faliures
16:39:46 mriedem https://bugs.launchpad.net/nova/+bug/1784093
16:39:46 openstack Launchpad bug 1784093 in OpenStack Compute (nova) "Build requests can be orphaned without instance mappings" [Medium,In progress] - Assigned to Mohammed Naser (mnaser)
16:39:59 mriedem ^ is about creating build requests, instance mappings and request specs in a single transaction
16:40:26 mriedem and then i was looking for a 2nd +2 on https://review.openstack.org/#/c/581910/ which is just documenting another up-call caveat
16:40:45 dansmith yeah, I think we also probably want a single transaction for converting a BR to a mapped mapping
16:41:04 mriedem i.e. create the instance in the cell + update the mapping?
16:41:14 mriedem how would that work across 2 dbs?
16:41:24 melwitt and destroying BR
16:41:27 mnaser my single transaction change, im not sure if it actually does it in a single transaction.. nor how to test it
16:41:45 melwitt mriedem: should be all api db, no? build request and instance mapping
16:41:46 mnaser i brought the requestspec, buildrequest and mapping to be a single transaction so that was a single db
16:42:25 dansmith mriedem: I mean the destroying the BR and setting cell_id on the instance mapping in one transaction
16:42:39 dansmith melwitt: right
16:43:26 mnaser i assume mriedem is concerned about updating instance mapping and creating instance in a single transaction
16:43:43 mnaser cause you can create but fail updating instance mapping
16:43:51 mnaser and end up with a listable but not retrievable instance
16:43:59 mriedem that's what https://review.openstack.org/#/c/586713/ is for
16:44:16 mriedem we update the mapping immediately after creating the instance,
16:44:25 mriedem but i'm not sure what should happen if we fail to update the mapping
16:44:32 dansmith destroy the instance
16:44:37 dansmith leave the BR in place
16:44:44 mriedem that's what it's doing right now
16:44:45 mriedem my patch
16:44:51 dansmith that's what we should do
16:44:57 melwitt mnaser: yeah, would have to take a look at what we can do with finding that out. jaypipes do you know how we can print/log whether something is occurring in a single db transaction when we're doing sqlalchemy queries while we're testing something?
16:45:00 mriedem but,
16:45:00 dansmith if we double fail on all DBs then we can't do anything anyway
16:45:12 mriedem if we have an empty mapping and no instance and a build request,
16:45:23 dansmith melwitt: one writer should mean one transaction AFAIK
16:45:24 mriedem the build request will always show as status BUILDING or whatever
16:45:53 mnaser thing is stale records in db that don't impact api aren't too much of a problem
16:46:19 mnaser having an instance mapping point to a cell where the instance doesnt exist => no confusing instance when listing instances
16:46:20 dansmith mriedem: the other thing we can do, which I'd rather not, but is look up mappings during a list and either filter those instances or fix the mappings on the fly
16:46:24 mriedem i don't think you can delete a scheduling instance though, unless you force delete it?
16:46:46 mriedem dansmith: i thought about fixing on the fly,
16:46:46 dansmith mriedem: I mean NFO the record you just created
16:46:47 mriedem however,
16:47:09 mriedem if we create the instance but fail to update the mapping, we won't have created the bdms and tags in the cell db either
16:47:11 melwitt dansmith: it should, but IIRC there are ways in which queries might not be a single transaction (from a db standpoint) even if you put them under the same decorator. it would be nice to be able to verify if we could
16:47:16 mriedem so the instance could be mapped, but it would be incomplete
16:47:42 dansmith melwitt: only if you're doing nested transactions or force commit, AFAIK
16:48:07 jaypipes melwitt: you mean can we inject log messages after running various DB statements in the placement service to log whether there were results?
16:48:16 mriedem i don't know what NFO means, urbandictionary suggests Near Fatal Orgasm
16:48:18 jaypipes melwitt: not quite sure what you're asking, sorry...
16:48:19 dansmith mriedem: sure, but that's going to affect the listable instance anyway
16:48:24 dansmith mriedem: nuke from orbit
16:48:40 melwitt jaypipes: I mean is there a way to log/print/see whether a series of queries we made really translated into a single database transaction?
16:48:43 dansmith mriedem: although I like yours better
16:48:51 mriedem probably gonna need to serialize these conversations for context
16:48:57 melwitt jaypipes: in general, this isn't about placement
16:49:02 mnaser as an operator i would rather have an instance_mapping that's invalid pointing to an non-existant instance (but not showing any instances in list) rather than instance showing in list and not retrievable
16:49:05 jaypipes melwitt: ah, you mean a single COMMIT was issued...
16:49:37 mriedem dansmith: maybe you want to just look at https://review.openstack.org/#/c/586713/ and leave whatever thoughts
16:49:46 jaypipes melwitt: that would prolly be something we could add to oslo.db's transaction context manager. Basically, output a LOG.info("Committing transaction") when the context manager completes without an exception?
16:49:50 mriedem oh you did
16:50:12 mnaser my single transaction change was updated here -- https://review.openstack.org/#/c/586742/ .. but no testing etc
16:50:24 gnuoy melwitt, sorry, just so I'm clear, there needs to be an [api_database] section in the neutron.conf that neutron-metadata-agent is using?
16:50:32 dansmith mnaser: I know, but there are several things we're trying to accomplish with the dance around creating/destroying those records
16:50:47 dansmith mnaser: mostly around trying to preserve api semantics that have existed for a long time
16:51:19 mnaser dansmith: makes sense, i dont have that much context about it then :p
16:51:35 dansmith mnaser: mriedem: maybe we should have a non-text conversation about that whole process to make sure we're still doing what we want
16:51:54 dansmith I'm not likely to be able to focus on it if we do it here, I just have too many parallel threads running
16:52:23 melwitt jaypipes: I guess so :) just curious if there's a way to sanity check that in a unit test or func test environment (temporarily), the sql commands that are issued to the database. I had thought that it's not guaranteed that everything you put under one transaction context manager is a single transaction, from a low-level database standpoint. maybe there's no way to check that? anyway, this is all just my curiosity, so sorry for the
16:52:23 melwitt distraction
16:53:19 mnaser dansmith: i agree :)
16:55:08 mriedem yeah, also,
16:55:29 melwitt gnuoy: I'm not 100% sure but I think no, the neutron-metadata-agent is a neutron thing and the nova part is a metadata proxy that sends to the agent
16:55:40 mriedem i'd kind of like to squash these 2 for loops in schedule_and_build_instance for sanity before making changes in here, but not sure how backportable that all is, since we likely need at least one fix going back to pike
16:56:06 mriedem mnaser: are you still on pike?
16:56:07 dansmith melwitt: FYI I'm not going to be around for the nova meeting tomorrow. I'm sure I will be sorely missed.
16:56:31 mnaser mriedem: no, queens everywhere at this point
16:56:34 melwitt dansmith: yes, we will prepare ourselves emotionally
16:56:47 gnuoy melwitt, yeah I assumed querying the nova api db would be a nova task which is why the guide seems odd when it says that neutron needs it.
16:57:25 melwitt gnuoy: well, the guide is saying the neutron metadata API proxy (emphasis on the proxy) needs it. the nova piece that proxies to the neutron agent needs it
16:57:44 dansmith gnuoy: melwitt: there is no [api_database] in the _neutron_ config
16:58:03 jaypipes melwitt: no need to apologize! it's an interesting problem.
16:58:59 gnuoy is the guide trying to tell me that nova-api-metadata service needs an [api_database] section?
17:01:00 melwitt dansmith: do you know if the metadata service and the neutron metadata API proxy the same service? their code is together in the tree

Earlier   Later